Evaluating Intermediate Reasoning of Code-Assisted Large Language Models for Mathematics

Dietrich Klakow; Nick Howell; Zena Al-Khalili

REVIEW 4 major objections 5 minor 36 references

Evaluating Intermediate Reasoning of Code-Assisted Large Language Models for Mathematics

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLMs can post high execution accuracy on math benchmarks even when their generated programs are logically unsound, so accuracy alone does not measure mathematical reasoning.

desk verdict A solid first look at the logic inside code-assisted math solutions, with a taxonomy worth using and a headline overstated by calling brute-force 'unsound.' read the letter →

arxiv 2504.17665 v2 pith:ELDCKOZD submitted 2025-04-24 cs.CL

Zena Al-Khalili , Nick Howell , Dietrich Klakow This is my paper · ORCID

classification cs.CL

keywords code-assistedLLMsmathematicalreasoninglogicalsoundnessprogramtaxonomyexecutionaccuracydecisiontreejudgeabstractsyntaxbenchmarkevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that execution accuracy on math benchmarks is a misleading measure of how well code-assisted LLMs actually reason. The authors analyze the Python programs that five LLMs generate as intermediate reasoning steps and classify each program as logically sound (grounded in a mathematical concept) or unsound (relying on brute-force search, memorized numbers, or no real logic at all). They find that models can achieve comparable execution accuracy regardless of whether their programs are sound, and that harder problems push all models toward unsound reasoning. Because soundness is what makes a generated solution verifiable and trustworthy, the paper concludes that code-assisted LLM evaluation must look beyond final-answer correctness.

What carries the argument

The paper's central object is a six-class taxonomy of generated programs — Conceptual, Primitive, From-scratch Implementation, Brute-Force, Disorganized, and No Logic — where the first three are considered logically sound because they ground the solution in a mathematical concept, and the last three are considered unsound. The taxonomy is operationalized by the Code-Structure Judge, a decision tree trained on structural features of each program's abstract syntax tree: counts of function calls, imports, built-in operations, control-flow statements, and variables that are defined-but-unused or used-but-undefined. This judge lets the authors scale a manual line-level annotation to all generated programs and then relate each program's logic class to its execution accuracy.

What would settle it

Run one model on matched easy/hard variants of the same word problem, with the prompt and demonstrations held fixed and only the numbers made larger, and measure the share of concept-grounded programs; if the hard variant does not shift the distribution toward brute-force and no-logic programs, the paper's difficulty effect is refuted.

Watch

Extended reading notes

Core claim

The central finding is that execution accuracy is decoupled from the logical soundness of the program an LLM generates. On the grade-school ASDiv set and the competition-level MATH500 set, the authors find that programs classified as Brute-Force or No Logic match the execution accuracy of concept-grounded programs, especially on the harder set. They also find that increasing problem difficulty decreases the share of sound generations for every evaluated model, including capable closed-source ones. To establish this, they build a six-class taxonomy of program logic and a decision-tree classifier, the Code-Structure Judge, that labels programs from abstract syntax tree features with 81% accuracy, outperforming an LLM-as-judge baseline at 73%.

Load-bearing premise

The paper's central finding hinges on its normative classification of logical soundness as 'grounding the implementation in a math concept,' a standard that labels brute-force search and direct-answer returns as unsound.

Editorial extensions

If this is right

Benchmark accuracy should no longer be read as evidence of mathematical reasoning quality, because unsound programs match sound ones in execution accuracy on hard problems.
As problem difficulty rises, even strong models shift from concept-grounded programs toward brute-force and no-logic solutions, meaning harder benchmarks can specifically reward reasoning hacks.
A correct execution outcome does not certify a trustworthy program; verifiability requires concept-grounded logic, not just the right answer.
Evaluations of code-assisted LLMs should report a soundness measure alongside accuracy, not accuracy alone.
Model capability shapes reasoning style: the more capable evaluated models ground more of their programs in math concepts, while the open-source models rely more on memorized and exhaustive-search approaches.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: If brute-force search were reclassified as logically sound — a finite exhaustive search is a legitimate algorithm — the headline finding would weaken to a statement about implementation style and efficiency rather than reasoning correctness.
Inference: The ASDiv-versus-MATH500 difficulty comparison is confounded by different prompts, demonstrations, and model subsets; a within-dataset matched-difficulty study could confirm or refute the difficulty effect.
Inference: The same taxonomy could be applied to natural-language chain-of-thought outputs, testing whether accuracy also hides unsound reasoning outside code.
Inference: Because the Code-Structure Judge has low recall on Disorganized programs (0.16), the reported soundness distributions contain non-trivial classification noise; a judge with semantic features would sharpen the quantitative claims.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. First, this is the first paper I've seen that looks inside the generated programs of code-assisted LLMs on math and classifies the reasoning style, rather than scoring only the final answer. The taxonomy (library call, primitive, from-scratch, brute-force, disorganized, no logic) is useful and the manual annotation is carefully done, with documented guidelines and high agreement on a double-annotated subset. Second, the central empirical observation holds in the data: execution accuracy can be high for programs that got the answer via a brute-force loop or a memorized constant. Table 1 shows Qwen2.5 at 100% brute-force on ASDiv and StarCoder2's No Logic accuracy above its Conceptual accuracy. That is a genuine problem for evaluation that stops at execution.

The soft spot is the paper's definition of 'unsound.' The authors define sound as 'grounding the implementation in a math concept,' which is a normative style preference, not logical validity. The GPT-4 LCM example in Figure 1 is a terminating, correct search for a common multiple; it would be fair to call it inefficient, but calling it unsound overstates the finding. The No Logic and Disorganized cases are genuinely unverifiable, and that alone is enough to support the paper's main recommendation—evaluation should look at more than execution accuracy. But the strong version of the claim, stated in the abstract and Section 5.2, depends on a definition that will not hold up under scrutiny.

There are two secondary issues. The difficulty comparison between ASDiv and MATH500 is confounded: different prompts, different numbers of demonstrations, and different model subsets, so 'harder problems lead to more unsound reasoning' is not cleanly identified. And there is no released code or data and no error bars on the per-class accuracies, which makes it harder to judge how stable the small-cell results are.

Who is this for? Anyone evaluating code-assisted LLMs on math or building evaluation metrics for reasoning. The taxonomy and the empirical picture are worth publishing, but the 'unsound' terminology should be softened and the difficulty analysis either re-run in a matched setting or reinterpreted. I'd send it to review—it deserves a serious referee—and the authors are honest about their limitations, which is a good sign.

Referee Report

4 major / 5 minor

Summary. The paper studies the Python programs that code-assisted large language models generate when solving math word problems, moving beyond execution accuracy to an analysis of the intermediate program logic. The authors propose a six-class taxonomy of program logic—Conceptual, Primitive, From-scratch, Brute-Force, Disorganized, and No Logic—and label the first three as logically sound and the last three as unsound. They manually annotate 300 programs, train a decision-tree classifier called Code-Structure Judge on AST-derived features, compare it with an o3-mini LLM judge, and apply the better judge to generations from five LLMs on ASDiv and MATH500. The main reported findings are that model capability affects the type of logic used, that the harder MATH500 problems reduce the share of sound programs, and that execution accuracy is comparable across sound and unsound classes, so accuracy-based evaluation hides reasoning-quality problems.

Significance. The paper makes a useful empirical contribution by drawing attention to program-level reasoning quality in code-assisted math solving, an area that is usually evaluated only by final-answer correctness. The manual annotation process is careful, with a per-line protocol and 93% line-level agreement, and the result that a small AST-feature decision tree outperforms an o3-mini LLM judge is an interesting and practical methodological finding. The taxonomy, while valuable as an analytical instrument, is built on a nonstandard definition of logical soundness, and several empirical claims are stated more strongly than the data support. If the terminology and claims are revised to target 'groundedness' or 'transparency' rather than logical soundness, the paper would be a solid contribution to more holistic evaluation of code-assisted LLMs.

major comments (4)

[Section 3.5; Figure 1; Table 1] The taxonomy labels Brute-Force programs as unsound because they do not ground the implementation in a math concept, but this is not logical unsoundness in the standard sense. A terminating exhaustive search is a valid algorithm: the GPT-4 while-loop in Figure 1 correctly computes the LCM for the given inputs, and it is verifiable and correct, though inefficient. Likewise, a No Logic program that directly returns a precomputed answer is not an unsound reasoning process; it is an absence of programmatic reasoning. Since Table 1 then shows that these 'unsound' classes often achieve accuracy comparable to the 'sound' classes, the central assertion in Section 5.2—that LLMs achieve comparable performance regardless of the logic they employ—partly conflates algorithmic style and efficiency with reasoning correctness.
[Section 5.2; Table 1] The claim that execution accuracy is comparable across all logic classes is not supported by the reported numbers. On ASDiv, GPT4 achieves 86% for Conceptual but 68% for Brute-Force and 63% for No Logic; GPT4o-mini achieves 90% for Conceptual but 77% for Brute-Force and 85% for No Logic. These are differences of up to 23 percentage points. No sample sizes per class, confidence intervals, or significance tests are reported, so the reader cannot tell whether the observed differences are real or noise. The data support a weaker claim: unsound or ungrounded classes can still achieve nontrivial accuracy, not that accuracy is comparable regardless of logic.
[Section 4.2; Figure 5] The comparison between ASDiv and MATH500 is confounded by more than problem difficulty. ASDiv uses the PAL prompt with three demonstrations, while MATH500 uses a different prompt with demonstrations from the MATH training split; GPT4o-mini and Qwen are evaluated in zero-shot settings on MATH500, and the model subsets differ between the two datasets. In addition, Figure 5 shows only GPT4o-mini, Qwen2.5, and StarCoder2, so the statement that increasing difficulty decreases sound generations 'for all models' is not supported for GPT4 and Llama3.1. The difficulty-driven conclusion should either be supported by a controlled comparison or restricted to the models and settings actually evaluated.
[Section 5.1; Table 1] The automated judge is the instrument behind the large-scale per-class accuracy results, but its reliability is uneven. Code-Structure Judge has 81% overall accuracy, yet the Disorganized class has recall of only 0.16 and precision of 0.50, and the paper acknowledges that the notably high accuracy of Disorganized programs in Table 1 is mainly due to false positives from the judge. Because per-class execution accuracy is load-bearing for the central claim, the paper should report the confusion-matrix-corrected estimates or explicitly exclude the least reliable classes from the accuracy comparison.

minor comments (5)

[Section 5.2 heading and text] The heading contains the typo 'Evauation', and the same section later contains 'Primitve' instead of 'Primitive'.
[Figure 5 caption] The caption should state why GPT4 and Llama3.1 are absent from the MATH500 logic distribution, since Figure 4 reports all five models on ASDiv.
[Table 1] The table header says 'Execution (macro) accuracy' but the paper does not define how the macro averaging is computed; please clarify whether accuracy is averaged per problem, per class, or per model.
[Section 3.1] The paper says programs that fail to parse are discarded, but it does not report how many programs were discarded or whether the discard rate varies by model; this information is needed to assess potential bias in the logic distributions.
[Throughout] Spelling of the taxonomy class is inconsistent: 'Disorganised' appears in Section 3.5 and some figures, while 'Disorganized' appears elsewhere; please standardize.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the accuracy–soundness comparison is measured from annotated programs, not derived from the paper's own definitions.

full rationale

The paper's central finding is an empirical measurement. Section 3.4 defines logical soundness as programs that "ground the implementation in a math concept," and Section 3.5 maps programs into six classes. The distributions over classes (Figures 4–5) and per-class execution accuracy (Table 1) are obtained by running the generated programs and comparing outputs to ground truth, not by algebraic manipulation of the definition. The strongest claim, that "Code-assisted LLMs can achieve comparable performance regardless of the type of logic they employ," is a reading of Table 1, where unsound classes such as Brute-Force and No Logic reach accuracies comparable to the sound classes; this is an empirical observation about the labeled data. The Code-Structure Judge is trained to reproduce human annotations, not to maximize or minimize accuracy differences, so the accuracy comparison is not fitted. No load-bearing self-citation or imported uniqueness theorem appears; citations such as PAL, ToRA, and Chen et al. (2025) are used for prompts or contextual observations, not to justify the paper's conclusion. The only caveat is interpretive: classifying exhaustive search as "unsound" is a stipulated definition, and the headline trustworthiness claim is conditional on that definition. This is a framing choice, not a circular derivation, and the paper is transparent about its definition and limitations, including the acknowledged weakness of the Code-Structure Judge on the Disorganized class.

Assumptions & free parameters 0 free parameters · 5 assumptions · 1 invented entities

The paper introduces no physical or mathematical free parameters. Its load-bearing assumptions are the normative definition of soundness, the exhaustiveness of the taxonomy, the reliability of human annotation, the sufficiency of AST features for the automated judge, and the validity of cross-dataset difficulty comparisons. These assumptions are acknowledged or partially validated, but none is proven.

assumptions (5)

ad hoc to paper A program is logically sound iff it grounds its implementation in a mathematical concept.
Defined in Section 3; this definition classifies brute-force and direct-answer programs as unsound, which is central to the paper's findings.
domain assumption The six taxonomy classes are mutually exclusive and cover all generated programs.
Section 3.5 proposes the taxonomy without a coverage proof; the decision tree judge assigns exactly one label.
domain assumption Human annotation is reliable.
They report 93% per-line and 100% per-sample agreement on 50 double-annotated samples, but the sample is small and no kappa coefficient is reported.
domain assumption The Code-Structure Judge's AST features are sufficient to classify soundness.
Section 3.6.2; supported only by 81% accuracy on a 90-example heldout set.
domain assumption ASDiv and MATH500 difficulty comparison is valid despite differing prompts and model subsets.
Section 5.2 compares distributions across datasets without controlling for prompt differences (PAL vs MATH prompt) or model availability (GPT4 only on ASDiv).

invented entities (1)

Taxonomy of six program logic classes (Conceptual, Primitive, From-scratch, Brute-force, Disorganized, No Logic)
purpose: To categorize the reasoning logic of generated programs.
The taxonomy is a new classification construct introduced in Section 3.5; its definitions determine the paper's findings and are not independently established or falsifiable outside the paper.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Evaluating Intermediate Reasoning of Code-Assisted Large Language Models for Mathematics." pith.science (2026). https://pith.science/paper/ELDCKOZD

@misc{pith2026250417665,
  author       = {Pith},
  title        = {Pith review of: Evaluating Intermediate Reasoning of Code-Assisted Large Language Models for Mathematics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ELDCKOZD}},
  note         = {Machine review of arXiv:2504.17665}
}

read the original abstract

Assisting LLMs with code generation improved their performance on mathematical reasoning tasks. However, the evaluation of code-assisted LLMs is generally restricted to execution correctness, lacking a rigorous evaluation of their generated programs. In this work, we bridge this gap by conducting an in-depth analysis of code-assisted LLMs generated programs in response to math reasoning tasks, with a focus on evaluating the soundness of the underlying reasoning processes. For this purpose, we assess the generations of five LLMs, on several math datasets, both manually and automatically, and propose a taxonomy of generated programs based on their logical soundness. Our findings show that the capabilities of models significantly impact the logic implemented to solve the problem. Closed-source LLMs ground their programs in mathematical concepts, whereas open-source models often resort to unsound reasoning, relying on memorized information and exhaustive searches. Furthermore, increasing the difficulty of problems decreases sound generations for all models, revealing a critical shortcoming of LLMs on complex mathematics, contrary to what accuracy metrics suggest. Our work highlights the need for more holistic evaluations of code-assisted LLMs beyond execution accuracy metrics, toward a better understanding of LLMs' limits in the math domain.

Figures

Figures reproduced from arXiv: 2504.17665 by the authors.

**Figure 2.** Cyclomatic complexity and API Calls of pro [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Confusion matrices, showing counts, of Code-Structure judge (left) and LLM-judge (right) on the held-out [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Distribution of programs logic for all evalu [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 6.** Figure 6: Distribution of programs logic, showing counts, per math subdomain in MATH500. Classes: ’FS Imp’: From-Scratch Implementation, ’BF’: BruteForce, ’Disorg’: Disorganized [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 7.** Figure 7: PAL prompt used to prompt evaluated models with three demonstrations on the ASDiv dataset. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

**Figure 9.** Figure 9: The prompt for the LLM-Judge task with o3-mini [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

**Figure 10.** Figure 10: Instances that were labeled “Disorganized” [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 3 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT -4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023
[4]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

arXiv 2021
[5]

Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. 2022. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588

arXiv 2022
[6]

Yongchao Chen, Harsh Jhamtani, Srinagesh Sharma, Chuchu Fan, and Chi Wang. 2025. https://openreview.net/forum?id=5X5Z7Ffrjb Steering large language models between code execution and textual reasoning . In The Thirteenth International Conference on Learning Representations

work page 2025
[7]

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://api.semanticscholar.org/CorpusID:239998651 Training verifiers to solve math word problems . ArXiv, abs/2110.14168

arXiv 2021
[8]

Debrup Das, Debopriyo Banerjee, Somak Aditya, and Ashish Kulkarni. 2024. Mathsensei: A tool-augmented large language model for mathematical reasoning. arXiv preprint arXiv:2402.17231

arXiv 2024

Show all 36 references

[9]

Shihan Dou, Haoxiang Jia, Shenxi Wu, Huiyuan Zheng, Weikang Zhou, Muling Wu, Mingxu Chai, Jessica Fan, Caishuang Huang, Yunbo Tao, Yan Liu, Enyu Zhou, Ming Zhang, Yuhao Zhou, Yueming Wu, Rui Zheng, Ming Wen, Rongxiang Weng, Jingang Wang, Xunliang Cai, Tao Gui, Xipeng Qiu, Qi Z...

2024
[10]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. License: Llama 3 Community License Agreement

2024 arXiv
[11]

Aryaz Eghbali and Michael Pradel. 2022. Crystalbleu: precisely and efficiently measuring the similarity of code. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, pages 1--12

2022
[12]

Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764--10799. PMLR

2023
[13]

Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2022. Roscoe: A suite of metrics for scoring step-by-step reasoning. arXiv preprint arXiv:2212.07919

2022 arXiv
[14]

Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen, et al. 2023. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint arXiv:2309.17452

2023 arXiv
[15]

Shibo Hao, Yi Gu, Haotian Luo, Tianyang Liu, Xiyan Shao, Xinyuan Wang, Shuhua Xie, Haodi Ma, Adithya Samavedhi, Qiyue Gao, et al. 2024. Llm reasoners: New evaluation, library, and analysis of step-by-step reasoning with large language models. arXiv preprint arXiv:2404.05221

2024 arXiv
[16]

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. arXiv preprint arXiv:2103.03874

2021 arXiv
[17]

Shima Imani, Liang Du, and Harsh Shrivastava. 2023. Mathprompter: Mathematical reasoning using large language models. arXiv preprint arXiv:2303.05398

2023 arXiv
[18]

Yeo Wei Jie, Ranjan Satapathy, Goh Siow Mong, Erik Cambria, et al. 2024. How interpretable are reasoning explanations from prompting large language models? arXiv preprint arXiv:2402.11863

2024 arXiv
[19]

Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Syste...

2022
[20]

Xiaoyuan Li, Wenjie Wang, Moxin Li, Junrong Guo, Yang Zhang, and Fuli Feng. 2024. Evaluating mathematical reasoning of large language models: A focus on error identification and correction. arXiv preprint arXiv:2406.00755

2024 arXiv
[21]

Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. https://arxiv.org/abs/2305.20050 Let's verify step by step . Preprint, arXiv:2305.20050

2023 arXiv
[22]

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36

2024
[23]

Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Z...

2024 arXiv
[24]

Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. Faithful chain-of-thought reasoning. arXiv preprint arXiv:2301.13379

2023 arXiv
[25]

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36

2024
[26]

Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2020. https://doi.org/10.18653/v1/2020.acl-main.92 A diverse corpus for evaluating and developing E nglish math word problem solvers . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pa...

2020 doi
[27]

Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

2025 arXiv
[28]

Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297

2020 arXiv
[29]

Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J \'e r \'e my Rapin, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950. License: Llama 2 Community License Agreement

2023 arXiv
[30]

Weixi Tong and Tianyi Zhang. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.1118 C ode J udge: Evaluating code generation with large language models . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20032--20051, Miami, Flori...

2024 doi
[31]

https://lyz-code.github.io/autoimport/ autoimport

Lyz lyz@riseup.net. https://lyz-code.github.io/autoimport/ autoimport . License: GPL-3.0
[32]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

2022
[33]

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

2020 doi
[34]

Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2023. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653

2023 arXiv
[35]

Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, et al. 2023 a . Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. arXiv preprint arXiv:2308.07921

2023 arXiv
[36]

Shuyan Zhou, Uri Alon, Sumit Agarwal, and Graham Neubig. 2023 b . Codebertscore: Evaluating code generation with pretrained models of code. arXiv preprint arXiv:2302.05527

2023 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT -4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023

[4] [4]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

arXiv 2021

[5] [5]

Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. 2022. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588

arXiv 2022

[6] [6]

Yongchao Chen, Harsh Jhamtani, Srinagesh Sharma, Chuchu Fan, and Chi Wang. 2025. https://openreview.net/forum?id=5X5Z7Ffrjb Steering large language models between code execution and textual reasoning . In The Thirteenth International Conference on Learning Representations

work page 2025

[7] [7]

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://api.semanticscholar.org/CorpusID:239998651 Training verifiers to solve math word problems . ArXiv, abs/2110.14168

arXiv 2021

[8] [8]

Debrup Das, Debopriyo Banerjee, Somak Aditya, and Ashish Kulkarni. 2024. Mathsensei: A tool-augmented large language model for mathematical reasoning. arXiv preprint arXiv:2402.17231

arXiv 2024