Pith. sign in

REVIEW 2 major objections 5 minor 7 cited by

HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation

T0 review · 2 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that current LLMs perform 10–15 percentage points worse on code-generation tasks that require the model to call its own earlier solution, and that instruction tuning barely narrows that gap.

desk verdict Harder code benchmark, but the self-invocation claim isn't valid without a call-structure check. read the letter →

arxiv 2412.21199 v2 pith:LZBA242B submitted 2024-12-30 cs.SE cs.CL

Zhaojian Yu , Yilun Zhao , Arman Cohan , Xiao-Ping Zhang This is my paper · ORCID
classification cs.SEcs.CL
keywords self-invokingcodegenerationbenchmarkHumanEvalProMBPPpass@1evaluationinstructiontuningchain-of-thoughtpromptingLLMreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces self-invoking code generation as a distinct task: the model first solves a base programming problem and then writes a second, harder function that must call the first function to complete. To study this, the authors build HumanEval Pro, MBPP Pro, and BigCodeBench-Lite Pro by generating more complex, semantically related variants of existing problems with verified canonical solutions and test cases. On these benchmarks, more than twenty models show a consistent 10–15 percentage-point pass@1 drop relative to the original HumanEval and MBPP, and the gap persists with one-shot prompting. The paper also finds that instruction-tuned models improve far less on self-invoking tasks than on standard ones, and that chain-of-thought prompting recovers only part of the loss. The result matters because it identifies a measurable capability—using one's own generated code as a building block—that current training and evaluation do not capture.

What carries the argument

The load-bearing object is the self-invoking code generation task, a paired-problem setup in which a model must solve a base problem and then a second, harder problem whose reference solution calls the first solution one or more times. The benchmark-construction pipeline generates the harder problems with a general-purpose code model, then iteratively executes candidate solutions with Python and manually reviews and fixes them until all canonical solutions pass the full test suite. Evaluation uses the standard pass@1 metric with the prompt instructing the model to write a Python file whose second solution requires single or multiple calls to the first, and error analysis classifies failures into assertion, name, value, index, type, and other runtime errors.

What would settle it

Modify the evaluation harness to log and inspect each generated program for a syntactic or semantic call to the base function before running hidden tests. If programs that contain no such call pass HumanEval Pro at a rate comparable to programs that do, the benchmark measures overall difficulty rather than self-invocation; if they fail, the phenomenon is real.

Watch

Extended reading notes

Core claim

The central claim is that current large language models are much better at generating a standalone function than at generating a related harder function that builds on it, and the paper calls this deficit "self-invoking code generation." The benchmarks operationalize the task by pairing a base problem with a self-invoking problem and asking the model to write both solutions in one response, with the second solution explicitly expected to make single or multiple calls to the first. The paper reports that o1-mini passes 96.2% of HumanEval but only 76.2% of HumanEval Pro, and that across the model family the decline is concentrated in failures on the harder self-invoking problems rather than in the base problems. It further claims that instruction-based fine-tuning, which lifts base models by large margins on HumanEval, gives only marginal gains on HumanEval Pro and MBPP Pro, suggesting that existing training recipes do not target this reuse capability.

Load-bearing premise

The scoring only runs final hidden tests, so a model that solves the harder problem without ever calling its base-solution function still passes; the benchmark therefore measures whether the model can solve a harder related problem, not necessarily whether it actually reuses its own code.

Editorial extensions

If this is right

  • Scores on HumanEval and MBPP overstate practical code generation ability, because a large share of the problems most models solve directly are not solved when the task requires composing a solution from the model's own code.
  • Instruction tuning as currently practiced is not sufficient for self-invoking code generation; closing this gap will require training data or objectives that specifically reward calling an earlier generated function.
  • Chain-of-thought prompting can be used as a partial mitigation: several models gain a few points on HumanEval Pro when asked to reason step by step, and the co-occurring drop in NameErrors suggests the reasoning makes generated code more self-contained.
  • The construction recipe is reproducible on other code benchmarks, as demonstrated by BigCodeBench-Lite Pro showing the same pattern, so the task can be extended as older benchmarks saturate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the authors did not run: instrument the evaluator to require, in addition to passing hidden tests, that the emitted code actually contains a call to the base function; this would separate "reuses own code" from "solves a harder related problem directly."
  • If self-invocation is the real bottleneck, then techniques that encourage decomposition—such as planning prompts, explicit function signatures, or training on compositional execution traces—should move HumanEval Pro scores more than further scaling of base-model data.
  • The observed NameError pattern suggests one concrete mechanism: models fail to carry the required function name or signature from the base problem into the second function, a determinable failure that could be fixed by consistency-aware decoding or prompt design.
  • Because the benchmarks are derived from a single generator model, part of the measured gap could be an artifact of how that generator phrases self-invoking problems; a multi-generator variant would show whether the 10–15 point drop is stable across problem phrasing.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper proposes a new code-generation task, self-invoking code generation, in which a model must solve a base problem and then solve a related, more complex problem by calling the function it wrote for the base problem. The authors generate three benchmarks, HumanEval Pro, MBPP Pro, and BigCodeBench-Lite Pro, by prompting DeepSeek-V2.5 to create harder variants of existing problems, then iteratively executing and manually reviewing canonical solutions and test cases. They evaluate more than twenty proprietary and open-source LLMs, reporting a consistent 10-15 point absolute pass@1 drop on the Pro benchmarks relative to their base counterparts (e.g., o1-mini at 96.2% on HumanEval versus 76.2% on HumanEval Pro), and they present error-type analyses and a chain-of-thought study. The central interpretation is that the performance gap measures LLMs' ability to invoke their own generated code, and that instruction-tuned models improve less on this ability than on standard code generation.

Significance. If the benchmarks validly require self-invocation, the paper makes a useful contribution: it provides a reproducible recipe for upgrading saturated benchmarks, public datasets and model checkpoints, consistent tables across three benchmark families, and a concrete failure-mode taxonomy. The reported drop is large and coherent across models, and the base-versus-instruct comparison is thought-provoking. However, the paper's central construct-validity claim is not yet established because the evaluation never verifies that generated code actually calls the base solution; without such verification, the observed gaps may simply reflect the higher difficulty of the Pro problems. The benchmark could still be valuable as a difficulty-scaled extension of HumanEval and MBPP even if the self-invocation interpretation is weakened, but the title, abstract, and Section 5 conclusions currently overstate what the experiments establish.

major comments (2)
  1. [Section 4 and Appendix F.2] The evaluation protocol only executes the final test cases and never inspects whether the model's second solution calls the first solution. A solution that inlines the base logic, defines a separate helper, or solves the harder problem directly passes the same assertions. The prompt instruction in Appendix F.2 ('requires single or multiple calls to the first solution') is not enforced by the harness. Consequently, the 10-15 point pass@1 drops in Table 2 and the confusion-matrix analysis in Section 5.2 may reflect overall problem difficulty rather than self-invocation ability. This is load-bearing for the paper's central claim, so I request a static or execution-based check that passing solutions actually invoke the base function, or a report of the fraction of passing solutions that contain such a call, with conclusions re-stated accordingly.
  2. [Section 5.1 and Eq. (1)] The claim that instruction-tuned models show 'marginal improvements' on self-invoking tasks rests on absolute point differences between HumanEval and HumanEval Pro (e.g., Qwen2.5-Coder-32B-instruct improves by 26.8 points on HumanEval but only 8.5 points on HumanEval Pro). Because the Pro problems are more difficult, absolute gains are not a clean measure of instruction-tuning efficacy, and the pattern is not uniform: Qwen2.5-Coder-1.5B-instruct loses 3.7 points on HumanEval Pro while gaining 26.8 points on HumanEval. The ratio defined in Eq. (1) is never used in the analysis. Please provide a difficulty-controlled or normalized comparison, and adjust the conclusion if the pattern is not robust.
minor comments (5)
  1. [Section 4] Specify exactly how pass@1 is computed for API models when temperature=0.2; the text says greedy decoding for open-source models and temperature=0.2 for API models, which leaves unclear the number of samples and whether the reported numbers are single-sample pass rates.
  2. [Table 2] The column header 'HumanEval (+)' is ambiguous; clarify which columns correspond to HumanEval, HumanEval+, MBPP, and MBPP+.
  3. [Appendix E] Using the line count of canonical solutions as a proxy for problem complexity is not validated; a more direct measure, such as the number of required calls or test-case difficulty, would better support the complexity claim.
  4. [Limitations] The Limitations section should acknowledge that the evaluation does not enforce the self-invocation constraint; currently it only mentions the Python-only restriction and the diversity of the generated problems.
  5. [Section 6.1] The choice of a 50%-70% solve-rate window for selecting BigCodeBench-Lite problems is not justified; please state the rationale and any sensitivity of the downstream results to this window.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the central pass@1 comparisons are self-contained and no derivation reduces to its inputs.

full rationale

The paper's reasoning is empirical rather than derivational: it constructs fixed benchmarks through a documented pipeline, evaluates more than twenty models with pass@1, and compares results against external HumanEval and MBPP baselines. No parameter is fitted to a subset of the data and then renamed as a prediction, no result is justified by a load-bearing self-citation chain, and no uniqueness theorem from the authors' prior work is invoked to forbid alternatives. The closest concern is that DeepSeek-V2.5 is both the benchmark generator and one of the evaluated models; this could bias that model's row, but it is a potential alignment issue rather than a circular derivation of the paper's main claims, which concern the general performance gap across many models. A second caveat is that the evaluator executes only final test cases and never verifies that generated code actually calls the base function, so the benchmark's construct validity as a measure of self-invocation is not fully established; however, that is a validity threat, not a circularity, because the observed 10%-15% pass@1 drop is measured directly from test execution and the analysis uses no fitted parameters. The benchmark construction, manual verification, and external-baseline comparisons make the paper self-contained under the circularity definition used here.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The benchmark construction leans on LLM-generated problems and tests plus hand-fixed canonical solutions. The main empirical claims do not rest on fitted numerical parameters, but they do depend on the assumptions that the tests are valid, that the evaluation enforces self-invocation, and that the selected BigCodeBench-Lite window is representative.

free parameters (2)
  • BigCodeBench-Lite solve-rate window = 50% to 70%
    Section 6.1: only problems with reported solve rate between 50% and 70% were selected for BigCodeBench-Lite Pro; this hand-chosen interval shapes the difficulty of the derived benchmark and the generalization claim.
  • Iteration stopping rule = 3 rounds
    Section 3.3 and Table 1: candidate solutions and test cases were iteratively fixed until a 100% pass rate on the canonical solution, a design choice that defines how the tests are reviewed.
assumptions (3)
  • domain assumption Generated test cases, after iterative execution and manual review, correctly capture the intended problem behavior
    Section 3.3: the 100% pass@1 table only verifies the canonical solutions against the tests; it does not validate that the tests reject incorrect solutions or match the problem statement for all valid programs.
  • domain assumption Passing the final test suite is a valid proxy for solving the self-invoking task, including the requirement to call the base solution
    Section 4 and Appendix F.2: the evaluation runs only assert-based tests and never inspects whether the generated code contains calls to the first function, so direct solutions are scored identically.
  • domain assumption The base problem and self-invoking problem are semantically linked such that solving the base is a necessary prerequisite
    Section 3.1 sets this as a guideline, but no verification is reported that the canonical or model solutions actually require the base function.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation." pith.science (2026). https://pith.science/paper/LZBA242B

@misc{pith2026241221199,
  author       = {Pith},
  title        = {Pith review of: HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LZBA242B}},
  note         = {Machine review of arXiv:2412.21199}
}
read the original abstract

We introduce self-invoking code generation, a new task designed to evaluate the progressive reasoning and problem-solving capabilities of LLMs. In this task, models are presented with a base problem and a related, more complex problem. They must solve the base problem and then utilize its solution to address the more complex one. This work features three key contributions. First, we propose a general recipe for generating more challenging versions of existing benchmarks, resulting in three new benchmarks: HumanEval Pro, MBPP Pro, and BigCodeBench-Lite Pro, specifically designed to assess LLMs on self-invoking code generation. Second, from the analysis of experimental results over twenty LLMs on our benchmarks, we have two important observations: (i) Most LLMs excel in traditional code generation benchmarks like HumanEval and MBPP, but their performance declines on self-invoking tasks. For example, o1-mini achieves 96.2% pass@1 on HumanEval but only 76.2% on HumanEval Pro. (ii) On self-invoking code generation task, the instruction-tuned models demonstrate only marginal improvements compared to the base models. Third, we disclose the types of failure modes that exist in our evaluation results. All these results underscore the need for further advancements in self-invoking code generation tasks and provide a new direction for future research on enhancing LLMs' code reasoning capabilities.

Figures

Figures reproduced from arXiv: 2412.21199 by the authors.

Figure 1
Figure 1. The overview of self-invoking code genera [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overview of benchmark construction. An example is shown in Figure [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Performance Comparison: HumanEval Pro (and MBPP Pro) vs. HumanEval (and MBPP). [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: HumanEval (or MBPP) scores against the results on HumanEval Pro and MBPP Pro (HumanEval+ and [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: The confusion matrix of different models. We use [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Error types of GPT-4o with and without CoT [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Statistics of error type across different LLMs [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: An example of self-invoking problems in HumanEval Pro [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Comparison between HumanEval Family, MBPP Family and BigCodeBench-Lite Family. [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Complexity comparison between base problem and self-invoking problem. We use the line count of [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Mode-local surrogate entropy asymmetrically reweights token advantages in RLVR, improving LLM math and code reasoning over SAPO, DAPO, and GTPO.

  2. Where Do LLMs Still Struggle? An In-Depth Analysis of Code Generation Benchmarks

    cs.SE 2025-11 conditional novelty 6.0 of 10

    Across four benchmarks and six LLMs, consistently failed code-generation tasks are driven by four recurring error patterns rather than by solution-code complexity.

  3. Can LLMs Generate High-Quality Test Cases for Algorithm Problems? TestCase-Eval: A Systematic Evaluation of Fault Coverage and Exposure

    cs.SE 2025-06 conditional novelty 6.0 of 10

    TestCase-Eval introduces fault coverage and fault exposure tasks over 500 Codeforces problems and finds that 19 LLMs, led by Qwen3-32B, remain far below human expert performance.

  4. AutoGEEval: A Multimodal and Automated Framework for Geospatial Code Generation on GEE with Large Language Models

    cs.SE 2025-05 conditional novelty 6.0 of 10

    An automated, execution-based benchmark of 1,325 Google Earth Engine unit tests shows 18 LLMs scoring between 31.40% and 71.55% pass@1, with parameter-knowledge errors the dominant failure mode.

  5. HackerRank-ASTRA: Evaluating Correctness & Consistency of Large Language Models on cross-domain multi-file project problems

    cs.LG 2025-01 conditional novelty 6.0 of 10

    A new 65-problem, multi-file benchmark evaluates LLM code generation on project-style tasks and reports that top models tie near 75%, with Claude-3.5-Sonnet most consistent.

  6. HiBayES: A Hierarchical Bayesian Modeling Framework for AI Evaluation Statistics

    cs.AI 2025-05 conditional novelty 5.0 of 10

    A hierarchical Bayesian GLM framework, HiBayES, provides uncertainty-aware performance estimates and formal model comparison for AI evaluation, with a software package and demonstrations on LLM benchmarks.

  7. Reasoning as a Resource: Optimizing Fast and Slow Thinking in Code Generation Models

    cs.SE 2025-06 conditional novelty 4.0 of 10

    Reasoning depth in code LLMs should be managed as a controllable resource across synthetic data generation, benchmarking, and deployment, rather than left implicit.

Reference graph

Works this paper leans on

44 extracted references · 9 canonical work pages · cited by 7 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    01.AI. 2024. Meet yi-coder: A small but mighty llm for code

    work page 2024

  4. [4]

    Anthropic. 2024. https://api.semanticscholar.org/CorpusID:268232499 The claude 3 model family: Opus, sonnet, haiku

    2024

  5. [5]

    Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, et al. 2022. Multi-lingual evaluation of code generation models. arXiv preprint arXiv:2210.14868

    arXiv 2022

  6. [6]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

    arXiv 2021

  7. [7]

    Antonio Valerio Miceli Barone and Rico Sennrich. 2017. A parallel corpus of python functions and documentation strings for automated code documentation and code generation. arXiv preprint arXiv:1707.02275

    arXiv 2017

  8. [8]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

Show all 44 references
  1. [9]

    DeepSeek-AI. 2024. https://arxiv.org/abs/2405.04434 Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model . Preprint, arXiv:2405.04434

    2024 arXiv

  2. [10]

    Alex Gu, Baptiste Roziere, Hugh James Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida Wang. 2024. Cruxeval: A benchmark for code reasoning, understanding and execution. In Forty-first International Conference on Machine Learning

    2024

  3. [11]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. Deepseek-coder: When the large language model meets programming--the rise of code intelligence. arXiv preprint arXiv:2401.14196

    2024 arXiv

  4. [12]

    Md Mahim Anjum Haque, Wasi Uddin Ahmad, Ismini Lourentzou, and Chris Brown. 2022. Fixeval: Execution-based evaluation of program fixes for competitive programming problems

    2022

  5. [13]

    Masum Hasan, Tanveer Muttaqueen, Abdullah Al Ishtiaq, Kazi Sajeed Mehrab, Md Mahim Anjum Haque, Tahmid Hasan, Wasi Uddin Ahmad, Anindya Iqbal, and Rifat Shahriyar. 2021. Codesc: A large code-description parallel dataset. arXiv preprint arXiv:2105.14220

    2021 arXiv

  6. [14]

    Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J Yang, JH Liu, Chenchen Zhang, Linzheng Chai, et al. 2024. Opencoder: The open cookbook for top-tier code large language models. arXiv preprint arXiv:2411.04905

    2024 arXiv

  7. [15]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186

    2024 arXiv

  8. [16]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint

    2024

  9. [17]

    Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. 2023. Impact of code language models on automated program repair. arXiv preprint arXiv:2302.05020

    2023 arXiv

  10. [18]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770

    2023 arXiv

  11. [19]

    Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, and Alexey Svyatkovskiy. 2023. Inferfix: End-to-end program repair with llms. arXiv preprint arXiv:2303.07263

    2023 arXiv

  12. [20]

    Jia Li, Ge Li, Xuanming Zhang, Yunfei Zhao, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, and Yongbin Li. 2024. https://openreview.net/forum?id=kvjbFVHpny Evocodebench: An evolving code generation benchmark with domain-specific evaluations . In The Thirty-eight Conference on Neu...

    2024

  13. [21]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161

    2023 arXiv

  14. [22]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36

    2024

  15. [23]

    Tianyang Liu, Canwen Xu, and Julian McAuley. 2023. Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint arXiv:2306.03091

    2023 arXiv

  16. [24]

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568

    2023 arXiv

  17. [25]

    Mistral. 2024. https://mistral.ai/news/codestral/ Codestral

    2024

  18. [26]

    Niklas Muennighoff, Qian Liu, Armel Randy Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. 2024. https://openreview.net/forum?id=mw1PWNSWZP Octopack: Instruction tuning code large language models . In The Twe...

    2024

  19. [27]

    Ansong Ni, Pengcheng Yin, Yilun Zhao, Martin Riddell, Troy Feng, Rui Shen, Stephen Yin, Ye Liu, Semih Yavuz, Caiming Xiong, Shafiq Joty, Yingbo Zhou, Dragomir Radev, Arman Cohan, and Arman Cohan. 2024. https://doi.org/10.1162/tacl_a_00705 L 2 CE val: Evaluating language-to-cod...

    2024 doi

  20. [28]

    OpenAI. 2024 a . https://openai.com/index/hello-gpt-4o Gpt-4o

    2024

  21. [29]

    OpenAI. 2024 b . https://api.semanticscholar.org/CorpusID:272648256 Openai o1 system card

    2024

  22. [30]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...

    2022

  23. [31]

    Houxing Ren, Mingjie Zhan, Zhongyuan Wu, Aojun Zhou, Junting Pan, and Hongsheng Li. 2024. https://arxiv.org/abs/2405.17057 Reflectioncoder: Learning from reflection sequence for enhanced one-off code generation . Preprint, arXiv:2405.17057

    2024 arXiv

  24. [32]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J \'e r \'e my Rapin, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950

    2023 arXiv

  25. [33]

    Disha Shrivastava, Hugo Larochelle, and Daniel Tarlow. 2023. Repository-level prompt generation for large language models of code. In International Conference on Machine Learning, pages 31693--31715. PMLR

    2023

  26. [34]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824--24837

    2022

  27. [35]

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2024. Magicoder: Empowering code generation with oss-instruct. In Forty-first International Conference on Machine Learning

    2024

  28. [36]

    Chunqiu Steven Xia, Yinlin Deng, and Lingming Zhang. 2024. Top leaderboard ranking = top coding proficiency, always? evoeval: Evolving coding benchmarks via llm. arXiv preprint

    2024

  29. [37]

    Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2022. Practical program repair in the era of large pre-trained language models. arXiv preprint arXiv:2210.14179

    2022 arXiv

  30. [38]

    Zhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang, Can Xu, Yishujie Zhao, Wenxiang Hu, and Qiufeng Yin. 2024. Wavecoder: Widespread and versatile enhancement for code large language models by instruction tuning. In Proceedings of the 62nd Annual Meeting of the Association for C...

    2024

  31. [39]

    Ziyin Zhang, Chaoyu Chen, Bingchang Liu, Cong Liao, Zi Gong, Hang Yu, Jianguo Li, and Rui Wang. 2023. https://arxiv.org/abs/2311.07989 A survey on language models for code

    2023 arXiv

  32. [40]

    Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, et al. 2023. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303.17568

    2023 arXiv

  33. [41]

    Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. 2024. Opencodeinterpreter: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402.14658

    2024 arXiv

  34. [42]

    Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipirneni, and Chandan K Reddy. 2022. Xlcost: A benchmark dataset for cross-lingual code intelligence. arXiv preprint arXiv:2206.08474

    2022 arXiv

  35. [43]

    Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. 2024. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931

    2024 arXiv

  36. [44]

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. 2024. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.