Pith. sign in

REVIEW 3 major objections 6 minor 10 cited by

Evaluating and Aligning CodeLLMs on Human Preference

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CodeArena measures code-LLM responses by human preference, not just correctness, and shows open models lag proprietary ones.

desk verdict A useful new code-preference benchmark and synthetic-data corpus, but the central claim that CodeArena measures human preference rests entirely on an unvalidated GPT-4o judge; the paper needs a human-validation study before the headline results can be trusted. read the letter →

arxiv 2412.05210 v1 pith:MWR7QZXW submitted 2024-12-06 cs.CL

Jian Yang , Jiaxi Yang , Ke Jin , Yibo Miao , Lei Zhang , Liqun Yang , Zeyu Cui , Yichang Zhang
show 2 more authors
Binyuan Hui Junyang Lin
This is my paper · ORCID
classification cs.CL
keywords CodeArenahumanpreferencealignmentLLMbenchmarkLLM-as-a-judgesyntheticinstructiondatagenerationQwen2.5-SynCoderwinrate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that execution-based code benchmarks miss a key axis of code assistant quality: whether responses match what real users prefer. To measure that axis, it introduces CodeArena, a set of 397 manually curated real-world coding questions spanning 40 categories and 44 programming languages, scored by pairwise GPT-4o judgments against a fixed baseline. Sweeping more than 40 models through CodeArena, it finds a substantial gap between open-source code LLMs such as Qwen2.5-Coder and proprietary models such as o1-mini and Claude-3.5-Sonnet, even where execution scores are close. The paper also builds SynCode-Instruct, a nearly 20-billion-token synthetic instruction corpus, and shows that training Qwen2.5-SynCoder on it improves both execution benchmarks and CodeArena win rates, though it does not close the preference gap.

What carries the argument

The load-bearing object is CodeArena itself: 397 human-curated user queries that deliberately resist unit-test evaluation and are scored by an LLM judge, GPT-4o-2024-08-06, in both presentation orders against a fixed baseline response from gpt-4-turbo-2024-04-09; the averaged win and tie rates are what make human preference alignment measurable. The training side rests on SynCode-Instruct, a nearly 20-billion-token synthetic instruction corpus generated by scaling questions from code Q&A websites, with algorithmic responses verified by generated test cases and non-algorithmic responses selected by an LLM scorer.

What would settle it

Run a human study in which developers rate the same 397 CodeArena responses that GPT-4o judged, and compute agreement between human and model verdicts; if agreement is near chance, or if GPT-4o's favorite responses are simply longer or more verbose rather than more useful, then CodeArena scores do not measure human preference.

Watch

Extended reading notes

Core claim

The central claim is that CodeArena measures alignment between model-generated responses and human preference, and that this alignment is a distinct axis from code correctness. On that axis, proprietary models dominate: o1-mini reaches an 89.3% win rate against the gpt-4-turbo-2024-04-09 baseline, Claude-3.5-Sonnet reaches 77.8%, and the best open model, Qwen2.5-Coder-32B-Instruct, reaches 68.9%, with most smaller open models far lower. The paper further claims that large-scale synthetic instruction data is an effective route to improving both execution scores and preference alignment, because Qwen2.5-SynCoder, trained entirely on the synthetic corpus, reaches 78.8% on MultiPL-E and 49.2% win rate on CodeArena, matching some much larger open models on execution while remaining below closed models on preference.

Load-bearing premise

The central claim leans on GPT-4o-2024-08-06 being a faithful judge of human preference, yet the paper provides no human preference labels, no inter-annotator agreement, and no direct correlation between GPT-4o verdicts and human verdicts.

Editorial extensions

If this is right

  • Execution benchmarks overstate how ready open-source code LLMs are for real user-facing assistance, because CodeArena exposes a response-quality deficit that Pass@1 does not capture.
  • The open-vs-proprietary gap on CodeArena points to a concrete failure mode: open models often emit code alone, while preferred responses include explanations, formatting, and comments.
  • Large-scale synthetic instruction data is a viable path to improve preference alignment in open models, as demonstrated by Qwen2.5-SynCoder's gains over its base.
  • Two-stage SFT, with synthetic data followed by a small high-quality set, outperforms one-stage training at the same token budget on both CodeArena and execution benchmarks.
  • CodeArena can serve as a dynamic leaderboard for tracking how well code LLMs align with human preference as new models are released.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If GPT-4o's verdicts track human preferences, CodeArena scores are a proxy for user satisfaction, but the paper offers no human-correlation check, so a crowdsourced validation study would directly test that link.
  • The preference gap may partly be a training-pipeline artifact: proprietary models are heavily optimized with human or AI preference feedback, and applying the same recipe to open models, with CodeArena as the reward signal, is a natural next step.
  • Because CodeArena queries are non-algorithmic and diverse, the benchmark could double as a general instruction-following test, not strictly a code benchmark.
  • The SynCode-Instruct pipeline inherits any bias of its LLM judge and scorer, so the same preferences being measured by CodeArena are also being baked into the training data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes CodeArena, a human-curated benchmark of 397 code-related questions extracted from real-world Q&A scenarios, spanning 40 categories and 44 programming languages, and evaluates models by pairwise comparison using GPT-4o-2024-08-06 as an LLM judge against a GPT-4-turbo baseline. It additionally introduces SynCode-Instruct, a roughly 20B-token synthetic instruction corpus derived from web data, and uses it to fine-tune Qwen2.5-SynCoder. The authors report CodeArena win rates for more than 40 models, execution-based results on EvalPlus and MultiPL-E, and conclude that open-source code LLMs lag proprietary models in human-preference alignment while large-scale synthetic instruction data improves execution performance.

Significance. If the central claim is validated, CodeArena would fill a genuine gap by providing a preference-oriented, real-world-distributed benchmark for code assistants, complementing correctness-only benchmarks such as HumanEval and MultiPL-E. The paper has concrete strengths: the questions are human-curated with a multi-stage quality-control process; the dataset covers a broad range of languages and task categories; the decontamination step against existing benchmarks is valuable; and the execution-based evaluations are straightforward and reproducible. The SynCode-Instruct corpus and the two-stage SFT recipe are also potentially useful community assets. However, the central claim that CodeArena measures alignment with human preference is currently unsupported, because the only preference signal is an unvalidated GPT-4o judge.

major comments (3)
  1. [Section 2, Evaluation; Section 5.1] The central claim that CodeArena measures alignment between model responses and human preference rests entirely on GPT-4o-2024-08-06 as the judge, but the paper provides no human preference labels, no inter-annotator agreement measure, and no correlation between GPT-4o verdicts and human verdicts. The paper's own Figure 1 illustrates that the judge favors 'a detailed and beautiful reply,' which suggests a possible length or explanation-style bias, and Table 3 shows that GPT-4o is itself an evaluated model with a 69.1% win rate, raising a self-preference concern that is not addressed. Without a human-validation study on a sample of CodeArena items, the reported open-vs-proprietary gap and all preference-alignment conclusions could be artifacts of the LLM judge. Please add a human evaluation with agreement statistics, or explicitly reframe CodeArena as measuring agreement with GPT-4o rather than with humans.
  2. [Abstract; Section 5.1; Table 3] The claim that Qwen2.5-SynCoder 'can achieve top-tier performance of open-source code LLMs' and that the results verify the effectiveness of large-scale synthetic instruction data for improving model performance is contradicted by the CodeArena results in Table 3: Qwen2.5-SynCoder achieves a 49.2% average win rate, well below Qwen2.5-Coder-32B-Instruct (68.9%) and Qwen2.5-72B-Instruct (73.8%), and the paper's Section 5.1 says only that SynCoder 'can still get a strong performance on CodeArena.' On the preference benchmark, the synthetic SFT recipe substantially degrades alignment relative to the strong base instruct model, yet this tension is never discussed. The 'top-tier' claim should be explicitly restricted to execution-based benchmarks, and the paper should explain or at least acknowledge the marked drop in CodeArena win rate for Qwen2.5-SynCoder.
  3. [Section 4.3; Section 5.1; Table 3] All CodeArena results are reported as point estimates from pairwise comparisons on 397 questions, with no confidence intervals, bootstrap estimates, or significance tests. Several headline comparisons in Table 3 are small (e.g., Claude-3.5-Sonnet-20240620 at 77.8% vs Claude-3.5-Sonnet-20241022 at 78.1%, or GPT-4o at 69.1% vs Qwen2.5-Coder-32B-Instruct at 68.9%) and are likely within sampling noise. The open-vs-proprietary gap is large, but the leaderboard ordering and the more fine-grained claims require uncertainty quantification. Please report bootstrap confidence intervals or pairwise significance tests for the main win rates.
minor comments (6)
  1. [Table 1] The category 'Miscellaneous and General Inquiry' appears twice with counts 62 and 7; rename the second occurrence to avoid ambiguity.
  2. [Table 3] The Qwen2.5-Coder-14B-Instruct row reports an average win/tie rate of 60.6/51.5, which is not derivable from the per-category values listed in the same row; please correct this entry and verify all other averages.
  3. [Abstract; Section 1; Section 7] The number of evaluated models is inconsistently reported as '40+ LLMs' in the abstract, 'nearly 40' in the introduction, and 'over 20' in the conclusion; please harmonize these counts.
  4. [Section 6] The final sentence of Related Work states 'In this work, we propose FullStack Bench,' but the paper proposes CodeArena; this appears to be a leftover from another manuscript and should be corrected.
  5. [Section 2, Human Annotation & Quality Control] The paper should clarify that the human annotation process validates question quality and difficulty, not the preference labels; the phrase 'HumanAnnotation ✓' in Table 2 could otherwise be misread as human preference annotation.
  6. [Section 3] The subsection heading 'Recall from Common Crawl' is incomplete as written; it should describe the data-source recall and cleaning pipeline more explicitly.

Circularity Check

1 steps flagged · score 4.0 of 10

CodeArena's central claim that it measures human preference rests entirely on GPT-4o judgments, with no human preference labels to break the equation; execution-benchmark results remain independent.

  1. self definitional [Section 2 'Evaluation' and Section 4.3 'LLM as a judgement'; central claim in Section 1 Contribution 3.]
    "Inspired by the previous work (Chiang et al., 2024), we apply GPT-4o-2024-08-06 as the judger to evaluate the model performance."

    CodeArena's only preference signal is GPT-4o pairwise verdicts, as stated in Section 2. The paper's central claim in Contribution 3 is that 'CodeArena can effectively measure the alignment between the model-generated response and human preference.' No human preference labels, inter-annotator agreement, or human-judge correlation appear anywhere in the paper. Therefore 'CodeArena measures human preference' is not empirically demonstrated but reduces by construction to 'CodeArena measures GPT-4o preference,' provided one equates the judge with human preference. The same judge also generated the second-stage SynCode-Instruct data, so the claimed preference alignment gain from that data is confounded with alignment to the judge.

full rationale

The paper contains no formal derivation, so the definitional circularity pattern is not present in equations. However, the central construct of the benchmark, 'human preference,' is operationalized exclusively by GPT-4o-2024-08-06 pairwise judgments, and the paper explicitly declines to collect human preferences citing cost. The claim that CodeArena measures human preference is thus equivalent to the untested assumption that GPT-4o's verdicts equal human preferences. This is a partial circularity because the target quantity is only observable through the same model used both to create high-quality training data and to score the benchmark. The execution-based evaluations on EvalPlus and MultiPL-E are externally grounded and independent, and the CodeArena questions themselves are human-curated, so the dataset construction is not circular. No load-bearing self-citation or imported uniqueness theorem was found. The score of 4 reflects a central claim that partially reduces to the judge's preferences, while other contributions retain independent content.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The benchmark rests on the trustworthiness of the LLM judge and the representativeness of website-sourced questions; both are domain assumptions rather than free parameters. No numbers are fitted to CodeArena outcomes, so the free_parameters list is empty.

assumptions (3)
  • domain assumption GPT-4o-2024-08-06 judgments are a faithful proxy for human preference in code Q&A.
    Section 2 Evaluation uses GPT-4o as the only judge; no human-labeled preference data or correlation analysis is reported. The paper's central human-preference claim depends on this.
  • domain assumption Online code Q&A sites and Common Crawl text are representative of real-world user coding queries.
    Sections 2 and 3 collect from websites; representativeness is assumed, not measured.
  • domain assumption Synthetic instruction data generated by Qwen2.5 and GPT-4o is diverse enough to improve alignment without degrading correctness.
    Section 3 and Figure 9 show scaling trends, but there is no human or external validation of alignment quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating and Aligning CodeLLMs on Human Preference." pith.science (2026). https://pith.science/paper/MWR7QZXW

@misc{pith2026241205210,
  author       = {Pith},
  title        = {Pith review of: Evaluating and Aligning CodeLLMs on Human Preference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MWR7QZXW}},
  note         = {Machine review of arXiv:2412.05210}
}
read the original abstract

Code large language models (codeLLMs) have made significant strides in code generation. Most previous code-related benchmarks, which consist of various programming exercises along with the corresponding test cases, are used as a common measure to evaluate the performance and capabilities of code LLMs. However, the current code LLMs focus on synthesizing the correct code snippet, ignoring the alignment with human preferences, where the query should be sampled from the practical application scenarios and the model-generated responses should satisfy the human preference. To bridge the gap between the model-generated response and human preference, we present a rigorous human-curated benchmark CodeArena to emulate the complexity and diversity of real-world coding tasks, where 397 high-quality samples spanning 40 categories and 44 programming languages, carefully curated from user queries. Further, we propose a diverse synthetic instruction corpus SynCode-Instruct (nearly 20B tokens) by scaling instructions from the website to verify the effectiveness of the large-scale synthetic instruction fine-tuning, where Qwen2.5-SynCoder totally trained on synthetic instruction data can achieve top-tier performance of open-source code LLMs. The results find performance differences between execution-based benchmarks and CodeArena. Our systematic experiments of CodeArena on 40+ LLMs reveal a notable performance gap between open SOTA code LLMs (e.g. Qwen2.5-Coder) and proprietary LLMs (e.g., OpenAI o1), underscoring the importance of the human preference alignment.\footnote{\url{https://codearenaeval.github.io/ }}

Figures

Figures reproduced from arXiv: 2412.05210 by the authors.

Figure 1
Figure 1. A comparison between the GPT4o with bet [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Task types of CodeArena. Difficulty levels of CodeArena [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Statistics of programming languages in CodeArena. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Number of samples of different difficulties (Easy/Medium/Hard) across categories in CodeArena. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Overview of the CodeArena creation benchmark. We first collect the online code Q&A and code-related [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Prompt of generating large-scale self￾contained synthetic instruction data. we test CodeLlama (Rozière et al., 2023), Open￾Coder (Huang et al., 2024), Qwen-Coder (Hui et al., 2024), DeepSeekCoder (Guo et al., 2024a), and CodeStral (MistralAI, 2024). 4.2 Evaluation Benc…
Figure 7
Figure 7. Figure 7: Examples of CodeArena. The LLM judger decides which response is better. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Comparison between MultiPL-E and CodeArena. LLMs in the blue circle present relatively mismatched performances on two benchmarks. tion (even the code is correct) will bring an unsatis￾factory experience to users, which will also lead to poor performance in CodeArena. I…
Figure 10
Figure 10. Figure 10: Distribution of CodeArena and MultiPL-E of different languages. whole area, while the representations of different languages in MultiPL-E are separately located in a narrow area. It shows that the distribution of queries in CodeArena is very diverse, which is suit￾abl…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Manner

    cs.CL 2025-06 conditional novelty 7.0 of 10

    SWE-Flow synthesizes incremental, test-driven development tasks from real GitHub projects and shows that fine-tuning Qwen2.5-Coder-32B-Instruct on them improves performance on the resulting SWE-Flow-Bench benchmark.

  2. IFEvalCode: Controlled Code Generation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A 1,620-sample, 8-language, Chinese/English benchmark separates code correctness from instruction-following and shows instruction compliance is far lower than correctness across 40+ LLMs.

  3. Turning the Tide: Repository-based Code Reflection

    cs.SE 2025-07 conditional novelty 6.0 of 10

    A new 1,888-case, six-language benchmark and a trained model measure and improve how code LLMs edit and repair code inside multi-file repositories.

  4. P2P: Automated Paper-to-Poster Generation and Fine-Grained Benchmark

    cs.CL 2025-05 conditional novelty 6.0 of 10

    P2P is a multi-agent framework that automatically generates HTML-rendered academic posters from papers, backed by a 30k instruction dataset and a 121-pair evaluation benchmark.

  5. LaQual: An Automated Framework for LLM App Quality Evaluation

    cs.SE 2025-08 reject novelty 5.0 of 10

    LaQual automates LLM app-store quality evaluation through scenario classification, static indicator filtering, and LLM-generated dynamic metrics, with Spearman correlations of about 0.6 against human ratings.

  6. Multi-Agent Collaboration for Multilingual Code Instruction Tuning

    cs.CL 2025-02 reject novelty 5.0 of 10

    A multi-agent collaboration framework for generating multilingual code instruction data is introduced, but the reported evaluations show the fine-tuned model underperforming its own base instruct model.

  7. BitsAI-CR: Automated Code Review via LLM in Practice

    cs.SE 2025-01 conditional novelty 5.0 of 10

    An industrial LLM-based code review system with a two-stage generate-and-filter pipeline and a data flywheel reached 75% precision and a 26.7% developer-action rate on Go code at ByteDance.

  8. Exploring the Implicit Semantic Ability of Multimodal Large Language Models: A Pilot Study on Entity Set Expansion

    cs.CL 2024-12 conditional novelty 5.0 of 10

    LUSAR applies listwise sampling and ranking to multimodal LLMs for entity set expansion and reports improved MESED scores, though the gains are confounded with supervised fine-tuning.

  9. ExecRepoBench: Multi-level Executable Code Completion Evaluation

    cs.CL 2024-12 reject novelty 5.0 of 10

    ExecRepoBench is an executable, repository-level code completion benchmark with unit tests, and Qwen2.5-Coder-Instruct-C is a 7B model fine-tuned on the accompanying Repo-Instruct corpus.

  10. MRT at SemEval-2025 Task 8: Maximizing Recovery from Tables with Multiple Steps

    cs.CL 2025-05 conditional novelty 4.0 of 10

    A code-generation pipeline with step-by-step instructions and error recovery scores 70.50% on the SemEval-2025 table question-answering task.

Reference graph

Works this paper leans on

53 extracted references · 11 canonical work pages · cited by 10 Pith papers

  1. [1]

    Anthropic. 2023. https://www.anthropic.com/index/introducing-claude Introducing Claude

    2023

  2. [2]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. https://arxiv.org/abs/2108.07732 Program synthesis with large language models . arXiv preprint arXiv:2108.07732

    arXiv 2021

  3. [3]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv 2023

  4. [4]

    Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. 2022. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207.14255

    arXiv 2022

  5. [5]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  6. [6]

    Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. 2023. https://doi.org/10.1109/TSE.2023.3267446 Multipl-e: A scalable and polyglot approach to benchmarking neural code generation . IEEE Trans...

    work page arXiv 2023

  7. [7]

    Linzheng Chai, Shukai Liu, Jian Yang, Yuwei Yin, Ke Jin, Jiaheng Liu, Tao Sun, Ge Zhang, Changyu Ren, Hongcheng Guo, et al. 2024. Mceval: Massively multilingual code evaluation. arXiv preprint arXiv:2406.07436

    arXiv 2024

  8. [9]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021 b . https://arxiv.org/abs/2107.03374 Evaluating large language models trained on code . ArXiv preprint, abs/2107.03374

    arXiv 2021

Show all 53 references
  1. [10]

    Jordan, Joseph E

    Wei - Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. https://openreview.net/forum?id=3MW8GKNyzI Chatbot arena: An open platform for evaluating l...

    2024

  2. [11]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  3. [12]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. https://doi.org/10.18653/V1/2020.FINDINGS-EMNLP.139 Codebert: A pre-trained model for programming and natural languages . In Findings of ...

    2020 doi

  4. [13]

    Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. 2023. https://openreview.net/forum?id=hQwb-lbM6EL Incoder: A generative model for code infilling and synthesis . In The Eleventh Internation...

    2023

  5. [14]

    Alex Gu, Baptiste Rozi \`e re, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. 2024. Cruxeval: A benchmark for code reasoning, understanding and execution

    2024

  6. [15]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. 2024 a . https://arxiv.org/abs/2401.14196 Deepseek-coder: When the large language model meets programming--the rise of code intelligence . arXiv preprint arXiv:2401.14196

    2024 arXiv

  7. [16]

    Jiawei Guo, Ziming Li, Xueling Liu, Kaijing Ma, Tianyu Zheng, Zhouliang Yu, Ding Pan, Yizhi Li, Ruibo Liu, Yue Wang, et al. 2024 b . Codeeditorbench: Evaluating code editing capability of large language models. arXiv preprint arXiv:2404.03543

    2024 arXiv

  8. [17]

    Junjie Huang, Duyu Tang, Linjun Shou, Ming Gong, Ke Xu, Daxin Jiang, Ming Zhou, and Nan Duan. 2021. Cosqa: 20,000+ web queries for code search and question answering

    2021

  9. [18]

    Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J Yang, JH Liu, Chenchen Zhang, Linzheng Chai, et al. 2024. Opencoder: The open cookbook for top-tier code large language models. arXiv e-prints, pages arXiv--2411

    2024

  10. [19]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186

    2024 arXiv

  11. [20]

    Faria Huq, Masum Hasan, Md Mahim Anjum Haque, Sazan Mahbub, Anindya Iqbal, and Toufique Ahmed. 2022. Review4repair: Code review aided automatic program repairing. 143:106765

    2022

  12. [21]

    Hamel Husain, Ho - Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. http://arxiv.org/abs/1909.09436 Codesearchnet challenge: Evaluating the state of semantic code search . arXiv preprint arXiv:1909.09436, abs/1909.09436

    2019 arXiv

  13. [22]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974

    2024 arXiv

  14. [23]

    Mingsheng Jiao, Tingrui Yu, Xuan Li, Guanjie Qiu, Xiaodong Gu, and Beijun Shen. 2023. On the evaluation of neural code translation: Taxonomy and benchmark. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 1529--1541. IEEE

    2023

  15. [24]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2015. http://arxiv.org/abs/1412.6980 Adam: A method for stochastic optimization . In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings

    2015 arXiv

  16. [25]

    Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming - Wei Chang, Andrew M

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming - Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and...

    2019 doi

  17. [26]

    Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-Tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2022. Ds-1000: A natural and reliable benchmark for data science code generation. ArXiv, abs/2211.11501

    2022 arXiv

  18. [27]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy - Poirier, Jo \ a o Mont...

    2023 doi arXiv

  19. [28]

    Ziming Li, Qianbo Zang, David Ma, Jiawei Guo, Tianyu Zheng, Xinyao Niu, Xiang Yue, Yue Wang, Jian Yang, Jiaheng Liu, et al. 2024. Autokaggle: A multi-agent framework for autonomous data science competitions. arXiv preprint arXiv:2410.20424

    2024 arXiv

  20. [29]

    Jiaheng Liu, Ken Deng, Congnan Liu, Jian Yang, Shukai Liu, He Zhu, Peng Zhao, Linzheng Chai, Yanan Wu, Ke Jin, et al. 2024 a . M2rc-eval: Massively multilingual repository-level code completion evaluation. arXiv preprint arXiv:2410.21157

    2024 arXiv

  21. [30]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. https://arxiv.org/abs/2305.01210 Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation . arXiv preprint arXiv:2305.01210, abs/2305.01210

    2023 arXiv

  22. [31]

    Shukai Liu, Linzheng Chai, Jian Yang, Jiajun Shi, He Zhu, Liran Wang, Ke Jin, Wei Zhang, Hualei Zhu, Shuyue Guo, et al. 2024 b . Mdeval: Massively multilingual code debugging. arXiv preprint arXiv:2411.02310

    2024 arXiv

  23. [32]

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, MING GONG, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shuj...

    2021

  24. [33]

    MistralAI. 2024. Codestral. https://mistral.ai/news/codestral. 2024.05.29

    2024

  25. [34]

    OpenAI. 2023. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . arXiv preprint arXiv:2303.08774

    2023 arXiv

  26. [35]

    Baptiste Rozi \`e re, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J \'e r \'e my Rapin, et al. 2023. https://arxiv.org/abs/2308.12950 Code Llama : Open foundation models for code . arXiv preprint arXiv:2308.12950

    2023 arXiv

  27. [36]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J \'e r \'e my Rapin, et al. 2023. Code llama: Open foundation models for code

    2023

  28. [37]

    Aofeng Su, Aowen Wang, Chao Ye, Chen Zhou, Ga Zhang, Guangcheng Zhu, Haobo Wang, Haokai Xu, Hao Chen, Haoze Li, et al. 2024. Tablegpt2: A large multimodal model with tabular data integration. arXiv preprint arXiv:2411.02059

    2024 arXiv

  29. [38]

    Tao Sun, Linzheng Chai, Yuwei Yin Jian Yang, Hongcheng Guo, Jiaheng Liu, Bing Wang, Liqun Yang, and Zhoujun Li. 2024. Unicoder: Scaling code large language model via universal code. ACL

    2024

  30. [39]

    Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. Debugbench: Evaluating debugging capability of large language models

    2024

  31. [40]

    Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11)

    2008

  32. [41]

    Xianjie Wu, Jian Yang, Linzheng Chai, Ge Zhang, Jiaheng Liu, Xinrun Du, Di Liang, Daixin Shu, Xianfu Cheng, Tianzhen Sun, et al. 2024. Tablebench: A comprehensive and complex benchmark for table question answering. arXiv preprint arXiv:2408.09174

    2024 arXiv

  33. [42]

    Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pages 1--10

    2022

  34. [43]

    Weixiang Yan, Yuchen Tian, Yunzhe Li, Qian Chen, and Wen Wang. 2023. https://aclanthology.org/2023.findings-emnlp.337 Codetransocean: A comprehensive multilingual benchmark for code translation . In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapo...

    2023

  35. [44]

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. 2024. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652

    2024 arXiv

  36. [45]

    Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. 2024. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. In Proceedings of the 46th IEEE/ACM International Conference on Sof...

    2024

  37. [46]

    Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. 2024. Mammoth2: Scaling instructions from the web. arXiv preprint arXiv:2405.03548

    2024 arXiv

  38. [47]

    Fengji Zhang, Bei Chen, Yue Zhang, Jin Liu, Daoguang Zan, Yi Mao, Jian - Guang Lou, and Weizhu Chen. 2023. https://doi.org/10.48550/arXiv.2303.12570 RepoCoder : Repository-level code completion through iterative retrieval and generation . arXiv preprint arXiv:2303.12570, abs/2...

    2023 doi arXiv

  39. [48]

    Shudan Zhang, Hanlin Zhao, Xiao Liu, Qinkai Zheng, Zehan Qi, Xiaotao Gu, Yuxiao Dong, and Jie Tang. 2024. Naturalcodebench: Examining coding performance mismatch on humaneval and natural user queries. In Findings of the Association for Computational Linguistics, ACL 2024, Bang...

    2024

  40. [49]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023 a . Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

  41. [50]

    Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. 2023 b . https://doi.org/10.48550/ARXIV.2303.17568 Codegeex: A pre-trained model for code generation with multilingual evaluations o...

    2023 doi arXiv

  42. [51]

    Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipirneni, and Chandan K Reddy. 2022. Xlcost: A benchmark dataset for cross-lingual code intelligence

    2022

  43. [52]

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. 2024. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877

    2024 arXiv

  44. [53]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  45. [54]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.