Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

IFEvalCode: Controlled Code Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that code LLMs pass tests far more often than they follow the user's style and structure constraints, and introduces a bilingual eight-language benchmark plus a training pipeline to measure and narrow that gap.

desk verdict Useful multilingual code benchmark with a plausible central gap, but the Instr. numbers ride on unvalidated regex checks and duplicated table rows — fix before citing. read the letter →

arxiv 2507.22462 v2 pith:ING4TAUL submitted 2025-07-30 cs.CL

Jian Yang , Wei Zhang , Shukai Liu , Linzheng Chai , Yingshui Tan , Jiaheng Liu , Ge Zhang , Wangchunshu Zhou
show 4 more authors
Guanglin Niu Zhoujun Li Binyuan Hui Junyang Lin
This is my paper
classification cs.CL
keywords controlledcodegenerationinstructionfollowingLLMevaluationmultilingualbenchmarkverifiableinstructionsconstraintsynthesisreject-samplingfine-tuningcorrectness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that today's code-generating models have a blind spot: they can produce code that passes unit tests far more often than code that obeys the explicit style, structure, and naming constraints a user asks for. To make that claim measurable, it builds IFEvalCode, a bilingual benchmark of 1,620 samples across eight programming languages, and scores every model twice: once on whether the generated code runs correctly and once on whether hand-written checkers confirm it followed the stated instructions. Across more than 40 models, the paper finds instruction-following scores run roughly 10 to 20 points below correctness, and even the strongest results reach only about 35% instruction compliance on English queries. The paper also proposes a data-generation pipeline, forward and backward constraint synthesis with reject-sampling fine-tuning, and reports that it lifts a 32B open model from 20.7% to 34.2% instruction compliance while improving correctness from 29.9% to 40.3%.

What carries the argument

The load-bearing piece is the check_instruction function, a hand-written, regex-based verifier attached to every benchmark sample that decides whether generated code obeys the stated constraints; for example, it certifies list comprehension by asserting that '[' and ']' appear in the code, and rules out built-in hash functions by checking that 'GetHashCode' is absent. These executable checks turn instruction-following into an objectively scored metric, Instr., separate from the unit-test-based correctness metric, Corr. The training side hinges on forward and backward constraints generation: forward generation picks constraint types and asks an LLM to create a question and both kinds of tests, while backward generation asks the LLM to write code first and then summarizes which constraints that code obeys, keeping only samples that pass both tests. Joint multilingual reject-sampling fine-tuning then optimizes the model for both answer generation and unit-test generation across all eight languages.

What would settle it

Take a random sample of IFEvalCode outputs, have independent human programmers judge whether each output truly obeys the stated constraints, and compare their verdicts with the check_instruction verdicts; if the disagreement rate is substantial, for example more than a few percent of cases where a checker passes code that a human reads as violating the constraint, or vice versa, then the Instr. numbers do not measure what the paper claims.

Watch

Extended reading notes

Core claim

The central discovery is that code controllability is a distinct and much harder capability than code correctness. On IFEvalCode, average instruction-following is consistently 10 to 20 points below correctness across model families, and no model comes close to satisfying all constraints: the best instruction-following scores sit near 35%, while correctness for the same models often passes 40% to 60% on individual languages. The gap holds in both Chinese and English, and the paper shows that correctness overlaps strongly across languages while instruction compliance does not, so obeying constraints does not automatically transfer when the prompt language changes. A further result is that the gap can be narrowed: fine-tuning on a corpus built by forward constraints generation, which writes questions from seed code plus chosen constraint types, and backward constraints generation, which writes questions from code and summarizes its constraints, filtered by reject sampling, raises both correctness and instruction-following on the benchmark.

Load-bearing premise

The Instr. scores are only as trustworthy as the hand-written regex checkers: if a checker passes code that violates the stated constraint, or rejects code that genuinely obeys it, then the reported 10 to 20 point gap between correctness and instruction-following is not an accurate measure of instruction-following.

Editorial extensions

If this is right

  • A benchmark that reports only unit-test pass rates overstates how usable a code model is for real coding tasks where formatting, size, and structural constraints are part of the request.
  • Instruction-following should be tracked as a separate number in code evaluation, not folded into correctness, because the two metrics behave differently across models, languages, and constraint types.
  • Synthesizing constraint-annotated training data with forward and backward generation, then filtering by execution, is a workable way to improve controllable code generation: it lifted one 32B model's instruction compliance from 20.7% to 34.2%.
  • Closed-source and larger models lead on controllable generation, but the gap is not mainly about scale: even the strongest models comply with only about a third of constraint sets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the measured gap is real, inference-time strategies that sample multiple candidates and filter by the check_instruction functions could close part of it, since those functions are executable and the paper uses similar filtering only during training.
  • The modest overlap between Chinese- and English-instruction compliance suggests constraint-following is partly a linguistic-formatting behavior; testing whether prompt-language-specific fine-tuning data improves both directions would be a natural follow-up.
  • The regex-based checkers could be replaced by stronger semantic verifiers, such as static analysis or LLM judges; if those agree with the current checkers, the 10 to 20 point gap is robust, and if not, the gap may be partially an artifact of how constraints are checked.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces IFEvalCode, a human-annotated bilingual benchmark for controlled code generation, covering 810 problems in 8 programming languages with paired Chinese and English queries and two separate check functions: one for code correctness (Corr.) and one for instruction-following (Instr.). The paper also proposes a forward/backward constraints generation pipeline to create an instruction corpus, trains a model called ControlledCoder via multilingual reject-sampling fine-tuning, and evaluates 40+ LLMs. The central empirical claim, stated in the abstract and introduction, is that LLMs' ability to generate instruction-compliant code is far weaker than their ability to generate correct code, with average Instr. scores roughly 10 to 20 points below Corr. scores across models and languages.

Significance. If the measurement is trustworthy, IFEvalCode addresses a real gap: existing code benchmarks mostly evaluate correctness, while real-world use often requires adherence to style, structural, and resource constraints. The decoupling of Corr. and Instr. is a useful and simple design, and the multilingual coverage (8 languages, paired Chinese/English queries) is a genuine contribution. The ControlledCoder training recipe is also plausible: the ablation in Table 5 shows consistent gains from forward constraints generation, backward constraints generation, and iterative reject sampling. However, the headline comparison between Corr. and Instr. rests on hand-written regex checks that have not been validated against human judgment, and the main results tables contain duplicate and internally inconsistent rows. These issues are load-bearing for the central claim and must be resolved before the empirical findings can be accepted.

major comments (4)
  1. [Section 2.2, Figure 3] The Instr. metric is computed by hand-written check_instruction functions, and these functions are never validated for precision or recall. Several examples in Figure 3 are demonstrably fragile: the Python check treats any presence of '[' and ']' as evidence of list comprehension (so a list literal or an indexing expression would pass); the Java check rejects any code containing the substring 'for' (so a comment or identifier such as 'format' would fail); and the C# check counts substring matches for LINQ method names rather than parsing actual method calls. These errors cut in both directions: loose checks inflate Instr. and shrink the reported gap, while strict checks deflate Instr. and inflate the gap. Because the central claim is a comparison between two metrics, the Instr. scorer must be validated. The paper reports no precision/recall, no inter-annotator agreement, and no manual audit of check_instruction outcomes. The quality-control description in Section 2.3 and Appendix D concerns question annotation, not the check functions themselves; Figure 4 labels the check-function step as 'LLM Refine' with no human validation. I ask for a human-validation study on a sample of check_instruction decisions, including adversarial counterexamples, and a report of the resulting precision/recall.
  2. [Table 3 and Table 4] The results tables contain identical rows for distinct models. In Table 3, the Qwen2.5-Coder-32B and Llama3.1-70B rows are identical; the Qwen3-32B and Qwen3-think-32B rows are identical; and the o1-mini, o3-mini, and o4-mini rows are identical. In Table 4, DeepSeek-R1's row is identical to its row in Table 3, as is the o1-mini row. While it is theoretically possible for two models to obtain the same rounded scores on 100 samples per language, the number of exact matches across all eight languages is implausible and suggests a copy/paste or evaluation-logging error. These rows must be corrected or explicitly explained, and the raw per-model results should be made available for verification.
  3. [Tables 3 and 4, per-language sample sizes] Each language sub-benchmark contains only about 100 problems, so per-cell scores are based on roughly 100 binary outcomes. For a score near 30%, the 95% binomial confidence interval is approximately ±9 percentage points; for scores near 50%, it is approximately ±10 points. Many of the differences used in the paper's rankings and in claims such as 'DeepSeek-R1 sets the highest group scores' are within this noise. For example, in Table 3, Python Corr. values of 58.1, 51.4, and 47.6 are not distinguishable at the per-language level. I request confidence intervals or significance tests for at least the headline comparisons, and I ask that the per-language rankings be softened or statistically justified.
  4. [Section 2.4 and Table 5] The check_correctness functions are generated by an LLM (Qwen2.5-Coder-32B) and then filtered only by whether the generated samples pass the unit tests. The paper does not report any manual audit of the correctness tests. If the LLM-generated unit tests are weak or incorrectly encoded, Corr. scores will be inflated, which would widen the reported Corr.-vs-Instr. gap. This is a second, parallel measurement risk that should be addressed, for example by reporting the number of LLM-generated tests that were rejected during quality control and by manually auditing a sample of check_correctness functions.
minor comments (6)
  1. [Section 2.2] The sentence defining the two check functions is reversed: the text says 'check_instruction' is used to check code correctness and 'check_correctness' judges instruction following, but the surrounding description and the rest of the paper make clear the opposite is intended.
  2. [Section 2.2 and Table 1] The text states 'Each question contains 3 constraints and 100 tokens on average,' but Table 1 reports an average of 5 instruction constraints and average question lengths of 340 and 358 tokens for English and Chinese. Please reconcile these numbers.
  3. [Abstract and Introduction] The abstract says IFEvalCode covers seven programming languages (Python, Java, JavaScript, TypeScript, Shell, C++, and C#), while the introduction, Table 1, and the rest of the paper list eight languages including PHP. The abstract should be corrected.
  4. [Figure 6] The Venn diagrams in Figure 6 are nearly illegible in the current rendering, and the caption does not state which model or models were used to generate the outputs. Please provide a high-resolution figure and specify the evaluation setup (model, temperature, number of samples) for this analysis.
  5. [Appendix F] The final paragraph of Appendix F is internally inconsistent with the rest of the paper: it states 'we propose IFEvalCode, a comprehensive code debugging benchmark covering 20 languages,' whereas this paper presents IFEvalCode as a code generation benchmark covering 8 languages. This appears to be leftover text from a different manuscript and should be removed or rewritten.
  6. [References] The reference list contains several duplicate entries: MultiPL-E appears as [9] and [10], HumanEval as [13] and [14], Qwen2.5-Coder Technical Report as [24], [25], and [26], MdEval as [38] and [39], and CodeLlama as [52] and [53]. Please deduplicate the bibliography.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the Corr./Instr. gap is an empirical measurement, not a derivation from the benchmark's own definitions.

full rationale

The paper's central claims are benchmark measurements, not derivations. IFEvalCode defines Corr. by execution against unit tests and Instr. by execution against hand-written check_instruction functions (Eq. 1; Sec. 2.2). The reported gap is an empirical contingency, not a logical consequence: the two metrics are independent, and several reported cells have Instr. > Corr. (e.g., grok-3 Python Instr. 29.5 vs Corr. 25.7; Qwen3 0.6B Instr. 23.8 vs Corr. 14.3), so the aggregate gap cannot be an artifact of the definitions. The benchmark was constructed by human annotators with explicit quality-control guidelines (Sec. 2.3), and the check functions in Fig. 3 are stated as hand-written rather than fitted to the reported model outputs. ControlledCoder is trained on a separately generated corpus (IFEvalCode-Instruct, Sec. 2.4) and evaluated on the human-annotated IFEvalCode; using Qwen2.5-Coder-32B both as a data generator and as the fine-tuning base is a same-family self-reference that could affect generalization, but the evaluation is external to the training samples and no fitted parameter is renamed as a prediction. The fragility of the regex check functions (e.g., the Python check treating the presence of '[' and ']' as evidence of list comprehension) is a measurement-validity concern, not circularity, because the paper does not use those checks to derive a theorem; it reports pass rates. No load-bearing self-citation, uniqueness import, or ansatz-by-citation appears in the derivation chain.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

Free parameters: the difficulty filter threshold (50% pass rate among strong LLMs) is a hand-chosen design decision that directly shapes the reported gap and model rankings; no other numeric parameters feed the central claim. The benchmark depends on domain assumptions about code extraction, regex-based constraint checking, test-set adequacy, translation fidelity, and non-contamination of the training corpus. No invented entities are introduced.

free parameters (1)
  • Difficulty filter threshold = 50% pass-rate cutoff among strong LLMs
    Section 2.3: annotators remove problems that half of GPT-4o, DeepSeek-V3, and Claude3.7 can answer. This hand-chosen threshold inflates task difficulty and the measured correctness-versus-instruction gap.
assumptions (5)
  • domain assumption Generated code can be extracted from model responses by locating a markdown code block; otherwise the full response is treated as code.
    Figure 3 check_instruction implementations use regex on triple-backtick blocks; models that omit fences are penalized.
  • domain assumption Hand-written regex check functions faithfully encode the human constraints and can detect violations (palindromic variables, banned loops, Greek lambda names, line counts, etc.).
    Section 2.2 and Figure 3; if these heuristics are bypassable or overly strict, Instr. scores mis-measure instruction following.
  • domain assumption The small unit-test suites (check_correctness) adequately capture functional correctness for each problem.
    Section 2.2; limited tests can mark incorrect code as correct and vice versa.
  • domain assumption Chinese queries are faithful translations of English queries such that the same check functions remain valid.
    Section 2.3; translation performed by annotators, no back-translation consistency check reported.
  • domain assumption The instruction corpus generated from Common Crawl code documents does not overlap with IFEvalCode benchmark problems, so training gains are not due to contamination.
    Section 2.4; no contamination analysis reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IFEvalCode: Controlled Code Generation." pith.science (2026). https://pith.science/paper/ING4TAUL

@misc{pith2026250722462,
  author       = {Pith},
  title        = {Pith review of: IFEvalCode: Controlled Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ING4TAUL}},
  note         = {Machine review of arXiv:2507.22462}
}
read the original abstract

Code large language models (Code LLMs) have made significant progress in code generation by translating natural language descriptions into functional code; however, real-world applications often demand stricter adherence to detailed requirements such as coding style, line count, and structural constraints, beyond mere correctness. To address this, the paper introduces forward and backward constraints generation to improve the instruction-following capabilities of Code LLMs in controlled code generation, ensuring outputs align more closely with human-defined guidelines. The authors further present IFEvalCode, a multilingual benchmark comprising 1.6K test samples across seven programming languages (Python, Java, JavaScript, TypeScript, Shell, C++, and C#), with each sample featuring both Chinese and English queries. Unlike existing benchmarks, IFEvalCode decouples evaluation into two metrics: correctness (Corr.) and instruction-following (Instr.), enabling a more nuanced assessment. Experiments on over 40 LLMs reveal that closed-source models outperform open-source ones in controllable code generation and highlight a significant gap between the models' ability to generate correct code versus code that precisely follows instructions.

Figures

Figures reproduced from arXiv: 2507.22462 by the authors.

Figure 1
Figure 1. Comparision between general code generation and Controllable code generation for 8 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2(a) shows domain types and Figure 2(b) lists instruction types. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Examples of the verifiable instructions with ‘ [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Construction of the benchmark IFEvalCode and instruction corpus IFEvalCode-Instruct. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Figure 5(a) and 5(b) plot the trends with the number of constraints increaing [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The Venn diagram of the numbers of the correct samples for each programming language. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Steerable Instruction Following Coding Data Synthesis with Actor-Parametric Schema Co-Evolution

    cs.SE 2026-02 unverdicted novelty 7.0 of 10

    IFCodeEvolve synthesizes coding data via actor-schema co-evolution with MCTS, boosting a 32B model's performance to match proprietary SOTA on instruction following.

  2. Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions?

    cs.CL 2025-09 conditional novelty 6.0 of 10

    A new 1,012-question benchmark shows LLMs often fail instructions that deliberately invert common training conventions, revealing a measurable gap in counterintuitive instruction following.

Reference graph

Works this paper leans on

81 extracted references · 25 canonical work pages · cited by 2 Pith papers

  1. [1]

    Introducing meta llama 3: The most capable openly available llm to date

    Meta AI. Introducing meta llama 3: The most capable openly available llm to date. https: //ai.meta.com/blog/meta-llama-3/, apr 2024

    2024

  2. [2]

    SantaCoder: Don’t reach for the stars! arXiv preprint arXiv:2301.03988, 2023

    Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Car- los Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. SantaCoder: Don’t reach for the stars! arXiv preprint arXiv:2301.03988, 2023

    arXiv 2023

  3. [3]

    Self-supervised bug de- tection and repair

    Miltiadis Allamanis, Henry Jackson-Flux, and Marc Brockschmidt. Self-supervised bug de- tection and repair. Advances in Neural Information Processing Systems , 34:27865–27876, 2021

    work page 2021

  4. [4]

    Introducing Claude, 2023

    Anthropic. Introducing Claude, 2023

    work page 2023

  5. [5]

    Claude 3.7 sonnet and claude code, 2025

    Anthropic. Claude 3.7 sonnet and claude code, 2025

    work page 2025

  6. [6]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  7. [7]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv 2023

  8. [8]

    Efficient training of language models to fill in the middle

    Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207.14255, 2022

    arXiv 2022

Show all 81 references
  1. [9]

    Multipl- e: A scalable and polyglot approach to benchmarking neural code generation.IEEE Transactions on Software Engineering, 2023

    Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. Multipl- e: A scalable and polyglot approach to benchmarking neural code generation.IEEE Transactions on S...

    2023

  2. [10]

    Multipl-e: A scalable and polyglot approach to benchmarking neural code generation

    Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. Multipl-e: A scalable and polyglot approach to benchmarking ne...

    2023

  3. [11]

    Mceval: Massively multilingual code evaluation

    Linzheng Chai, Shukai Liu, Jian Yang, Yuwei Yin, Ke Jin, Jiaheng Liu, Tao Sun, Ge Zhang, Changyu Ren, Hongcheng Guo, et al. Mceval: Massively multilingual code evaluation. arXiv preprint arXiv:2406.07436, 2024

    2024 arXiv

  4. [12]

    Coder: Issue resolving with multi-agent and task graphs

    Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian-Gang Wang, Anton Cheshkov, Jun Sun, Hao Yu, Guoliang Dong, Artem Aliev, et al. Coder: Issue resolving with multi-agent and task graphs. arXiv preprint arXiv:2406.01304, 2024

    2024 arXiv

  5. [14]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. ArXiv preprint, abs/2107.03374, 2021

    2021 arXiv

  6. [15]

    Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models

    Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. In Proceedings of the 32nd ACM SIGSOFT international symposium on software testing and analy...

    2023

  7. [16]

    BERT: pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu...

    2019

  8. [17]

    Codebert: A pre-trained model for programming and natural languages

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. Codebert: A pre-trained model for programming and natural languages. In Trevor Cohn, Yulan He, and Yang Liu, editors,Findings of the Associatio...

    2020

  9. [18]

    Gemma: Open models based on gemini research and technology

    Google Gemma Team. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  10. [19]

    The counterfeit conundrum: Can code language models grasp the nuances of their incorrect generations? arXiv preprint arXiv:2402.19475, 2024

    Alex Gu, Wen-Ding Li, Naman Jain, Theo X Olausson, Celine Lee, Koushik Sen, and Armando Solar-Lezama. The counterfeit conundrum: Can code language models grasp the nuances of their incorrect generations? arXiv preprint arXiv:2402.19475, 2024

    2024 arXiv

  11. [20]

    Cruxeval: A benchmark for code reasoning, understanding and execution

    Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. 2024

    2024

  12. [21]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  13. [22]

    Deepseek-coder: When the large language model meets program- ming – the rise of code intelligence

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. Deepseek-coder: When the large language model meets program- ming – the rise of code intelligence. arXiv preprint arXiv:2401.14196, 2024

    2024 arXiv

  14. [23]

    Opencoder: The open cookbook for top-tier code large language models

    Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J Yang, JH Liu, Chenchen Zhang, Linzheng Chai, et al. Opencoder: The open cookbook for top-tier code large language models. arXiv preprint arXiv:2411.04905, 2024

    2024 arXiv

  15. [26]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jia- jun Zhang, Bowen Yu, Kai Dang, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186, 2024. 11

    2024 arXiv

  16. [27]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  17. [28]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  18. [29]

    Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    2023 arXiv

  19. [30]

    xcodeeval: A large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval

    Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. xcodeeval: A large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval. arXiv preprint arXiv:2303.03004, 2023

    2023 arXiv

  20. [31]

    Wang, and Tao Yu

    Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen- Tau Yih, Daniel Fried, Sida I. Wang, and Tao Yu. DS-1000: A natural and reliable benchmark for data science code generation. InInternational Conference on Machine Learning, ICML 2023, 23-29 ...

    2023

  21. [32]

    Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161, abs/2305.06161, 2023

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozh- skii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro...

    2023 arXiv

  22. [33]

    Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, ...

    2022 arXiv

  23. [34]

    Autokaggle: A multi-agent framework for autonomous data science competitions

    Ziming Li, Qianbo Zang, David Ma, Jiawei Guo, Tianyu Zheng, Xinyao Niu, Xiang Yue, Yue Wang, Jian Yang, Jiaheng Liu, et al. Autokaggle: A multi-agent framework for autonomous data science competitions. arXiv preprint arXiv:2410.20424, 2024

    arXiv

  24. [35]

    Quixbugs: a multi- lingual program repair benchmark set based on the quixey challenge

    Derrick Lin, James Koppel, Angela Chen, and Armando Solar-Lezama. Quixbugs: a multi- lingual program repair benchmark set based on the quixey challenge. InProceedings Companion of the 2017 ACM SIGPLAN international conference on systems, programming, languages, and application...

    2017

  25. [36]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  26. [37]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. arXiv preprint arXiv:2305.01210, abs/2305.01210, 2023. 12

    2023 arXiv

  27. [39]

    Mdeval: Massively multilingual code debugging

    Shukai Liu, Linzheng Chai, Jian Yang, Jiajun Shi, He Zhu, Liran Wang, Ke Jin, Wei Zhang, Hualei Zhu, Shuyue Guo, et al. Mdeval: Massively multilingual code debugging. arXiv preprint arXiv:2411.02310, 2024

    2024 arXiv

  28. [40]

    Fullstack bench: Evaluating llms as full stack coder

    Siyao Liu, He Zhu, Jerry Liu, Shulin Xin, Aoyan Li, Rui Long, Li Chen, Jack Yang, Jinxiang Xia, ZY Peng, et al. Fullstack bench: Evaluating llms as full stack coder. arXiv preprint arXiv:2412.00535, 2024

    2024 arXiv

  29. [41]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  30. [42]

    Starcoder 2 and the stack v2: The next generation

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Noua- mane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173, 2024

    2024 arXiv

  31. [43]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  32. [44]

    Introducing gpt-4.5, 2025

    OpenAI. Introducing gpt-4.5, 2025

    2025

  33. [45]

    Measuring the impact of programming language distribution

    Gabriel Orlanski, Kefan Xiao, Xavier Garcia, Jeffrey Hui, Joshua Howland, Jonathan Malmaud, Jacob Austin, Rishabh Singh, and Michele Catasta. Measuring the impact of programming language distribution. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara En- gelhardt, Siva...

    2023

  34. [46]

    Humaneval-xl: A multilingual code generation benchmark for cross-lingual natural language generalization

    Qiwei Peng, Yekun Chai, and Xuhong Li. Humaneval-xl: A multilingual code generation benchmark for cross-lingual natural language generalization. arXiv preprint arXiv:2402.16694, 2024

    2024 arXiv

  35. [47]

    Deepbugs: A learning approach to name-based bug detection

    Michael Pradel and Koushik Sen. Deepbugs: A learning approach to name-based bug detection. Proceedings of the ACM on Programming Languages, 2(OOPSLA):1–25, 2018

    2018

  36. [48]

    Can openai’s codex fix bugs? an evaluation on quixbugs

    Julian Aron Prenner, Hlib Babii, and Romain Robbes. Can openai’s codex fix bugs? an evaluation on quixbugs. In Proceedings of the Third International Workshop on Automated Program Repair, pages 69–75, 2022

    2022

  37. [49]

    Runbugrun – an executable dataset for automated program repair

    Julian Aron Prenner and Romain Robbes. Runbugrun – an executable dataset for automated program repair. arXiv preprint arXiv:2304.01102, 2023

    2023 arXiv

  38. [50]

    Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings

    Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, et al. Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings. arXiv preprint arXiv:2501.01257, 2025

    2025 arXiv

  39. [51]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. OpenAI blog, 2018

    2018

  40. [52]

    Code llama: Open foundation models for code

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  41. [53]

    Code llama: Open foundation models for code

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. 2023

    2023

  42. [54]

    Bloom: A 176b-parameter open-access multilingual language model

    Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili ´c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. 13

    2022 arXiv

  43. [55]

    An analysis of the automatic bug fixing performance of chatgpt

    Dominik Sobania, Martin Briesch, Carol Hanna, and Justyna Petke. An analysis of the automatic bug fixing performance of chatgpt. In 2023 IEEE/ACM International Workshop on Automated Program Repair (APR), pages 23–30. IEEE, 2023

    2023

  44. [56]

    Tablegpt2: A large multimodal model with tabular data integration

    Aofeng Su, Aowen Wang, Chao Ye, Chen Zhou, Ga Zhang, Guangcheng Zhu, Haobo Wang, Haokai Xu, Hao Chen, Haoze Li, et al. Tablegpt2: A large multimodal model with tabular data integration. arXiv preprint arXiv:2411.02059, 2024

    2024 arXiv

  45. [57]

    UniCoder: Scaling code large language model via universal code

    Tao Sun, Linzheng Chai, Jian Yang, Yuwei Yin, Hongcheng Guo, Jiaheng Liu, Bing Wang, Liqun Yang, and Zhoujun Li. UniCoder: Scaling code large language model via universal code. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of ...

    2024

  46. [58]

    Bugs in large language models generated code: An empirical study

    Florian Tambon, Arghavan Moradi Dakhel, Amin Nikanjam, Foutse Khomh, Michel C Des- marais, and Giuliano Antoniol. Bugs in large language models generated code: An empirical study. CoRR, 2024

    2024

  47. [59]

    Magis: Llm-based multi-agent framework for github issue resolution

    Wei Tao, Yucheng Zhou, Wenqiang Zhang, and Yu Cheng. Magis: Llm-based multi-agent framework for github issue resolution. arXiv preprint arXiv:2403.17927, 2024

    2024 arXiv

  48. [60]

    Debugbench: Evaluating debugging capability of large language models

    Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Debugbench: Evaluating debugging capability of large language models. arXiv preprint arXiv:2401.04621, 2024

    2024 arXiv

  49. [61]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  50. [62]

    Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859, 2021

    2021 arXiv

  51. [63]

    Where do large language models fail when generating code? arXiv preprint arXiv:2406.08731, 2024

    Zhijie Wang, Zijie Zhou, Da Song, Yuheng Huang, Shengmai Chen, Lei Ma, and Tianyi Zhang. Where do large language models fail when generating code? arXiv preprint arXiv:2406.08731, 2024

    2024 arXiv

  52. [64]

    Fixing code generation errors for large language models

    Hao Wen, Yueheng Zhu, Chao Liu, Xiaoxue Ren, Weiwei Du, and Meng Yan. Fixing code generation errors for large language models. arXiv preprint arXiv:2409.00676, 2024

    2024 arXiv

  53. [65]

    Tablebench: A comprehensive and complex benchmark for table question answering

    Xianjie Wu, Jian Yang, Linzheng Chai, Ge Zhang, Jiaheng Liu, Xinrun Du, Di Liang, Daixin Shu, Xianfu Cheng, Tianzhen Sun, et al. Tablebench: A comprehensive and complex benchmark for table question answering. arXiv preprint arXiv:2408.09174, 2024

    2024 arXiv

  54. [66]

    Fuzz4all: Universal fuzzing with large language models

    Chunqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. Fuzz4all: Universal fuzzing with large language models. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–13, 2024

    2024

  55. [67]

    Conversational automated program repair

    Chunqiu Steven Xia and Lingming Zhang. Conversational automated program repair. arXiv preprint arXiv:2301.13246, 2023

    2023 arXiv

  56. [68]

    CodeTransOcean: A comprehensive multilingual benchmark for code translation

    Weixiang Yan, Yuchen Tian, Yunzhe Li, Qian Chen, and Wen Wang. CodeTransOcean: A comprehensive multilingual benchmark for code translation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5067–...

    2023

  57. [69]

    Evaluating and aligning codellms on human preference

    Jian Yang, Jiaxi Yang, Ke Jin, Yibo Miao, Lei Zhang, Liqun Yang, Zeyu Cui, Yichang Zhang, Binyuan Hui, and Junyang Lin. Evaluating and aligning codellms on human preference. arXiv preprint arXiv:2412.05210, 2024

    2024 arXiv

  58. [70]

    Fuzzcoder: Byte-level fuzzing test via large language model

    Liqun Yang, Jian Yang, Chaoren Wei, Guanglin Niu, Ge Zhang, Yunli Wang, Linzheng ChaI, Wanxu Xia, Hongcheng Guo, Shun Zhang, et al. Fuzzcoder: Byte-level fuzzing test via large language model. arXiv preprint arXiv:2409.01944, 2024. 14

    2024 arXiv

  59. [71]

    Enhancing the code debugging ability of llms via communicative agent based data refinement

    Weiqing Yang, Hanbin Wang, Zhenghao Liu, Xinze Li, Yukun Yan, Shuo Wang, Yu Gu, Minghe Yu, Zhiyuan Liu, and Ge Yu. Enhancing the code debugging ability of llms via communicative agent based data refinement. arXiv preprint arXiv:2408.05006, 2024

    2024 arXiv

  60. [72]

    Natural language to code generation in interactive data science notebooks

    Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Oleksandr Polozov, and Charles Sutton. Natural language to code generation in interactive data science notebooks. In Anna Rogers, Jo...

    2023

  61. [73]

    Codereval: A benchmark of pragmatic code generation with generative pre-trained models

    Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. In Proceedings of the 46th IEEE/ACM International Conference on Software ...

    2024

  62. [74]

    Evaluating instruction-tuned large language models on code comprehension and generation

    Zhiqiang Yuan, Junwei Liu, Qiancheng Zi, Mingwei Liu, Xin Peng, and Yiling Lou. Evaluating instruction-tuned large language models on code comprehension and generation. arXiv preprint arXiv:2308.01240, 2023

    2023 arXiv

  63. [75]

    Mammoth2: Scaling instructions from the web

    Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web. arXiv preprint arXiv:2405.03548, 2024

    2024 arXiv

  64. [76]

    Prompt-enhanced software vulnerability detection using chatgpt

    Chenyuan Zhang, Hao Liu, Jiutian Zeng, Kejing Yang, Yuhong Li, and Hui Li. Prompt-enhanced software vulnerability detection using chatgpt. In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, pages 276–277, 2024

    2024

  65. [77]

    RepoCoder: Repository-level code completion through iterative retrieval and generation

    Fengji Zhang, Bei Chen, Yue Zhang, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. RepoCoder: Repository-level code completion through iterative retrieval and generation. arXiv preprint arXiv:2303.12570, abs/2303.12570, 2023

    2023 arXiv

  66. [78]

    A critical review of large language model on software engineering: An example from chatgpt and automated program repair

    Quanjun Zhang, Tongke Zhang, Juan Zhai, Chunrong Fang, Bowen Yu, Weisong Sun, and Zhenyu Chen. A critical review of large language model on software engineering: An example from chatgpt and automated program repair. arXiv preprint arXiv:2310.08879, 2023

    arXiv

  67. [79]

    Naturalcodebench: Examining coding performance mismatch on humaneval and natural user prompts

    Shudan Zhang, Hanlin Zhao, Xiao Liu, Qinkai Zheng, Zehan Qi, Xiaotao Gu, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. Naturalcodebench: Examining coding performance mismatch on humaneval and natural user prompts. arXiv preprint arXiv:2405.04520, 2024

    2024 arXiv

  68. [80]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    2023

  69. [82]

    Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x

    Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303.17568, abs...

    2023 arXiv

  70. [83]

    Kun: Answer polishment for chinese self-alignment with instruction back-translation

    Tianyu Zheng, Shuyue Guo, Xingwei Qu, Jiawei Guo, Weixu Zhang, Xinrun Du, Chenghua Lin, Wenhao Huang, Wenhu Chen, Jie Fu, et al. Kun: Answer polishment for chinese self-alignment with instruction back-translation. arXiv preprint arXiv:2401.06477, 2024

    2024 arXiv

  71. [84]

    Advancing bug detection in fastjson2 with large language models driven unit test generation

    Zhiyuan Zhong, Sinan Wang, Hailong Wang, Shaojin Wen, Hao Guan, Yida Tao, and Yepang Liu. Advancing bug detection in fastjson2 with large language models driven unit test generation. arXiv preprint arXiv:2410.09414, 2024

    2024

  72. [85]

    Instruction-following evaluation for large language models

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. 15

    2023 arXiv

  73. [86]

    use camelCase

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Bench- marking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15...

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.