Pith. sign in

REVIEW 3 major objections 5 minor 104 references

Code LLMs: A Taxonomy-based Survey

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This survey proposes a five-part taxonomy to organize Code LLM research into tasks, corpora, models, benchmarks, and challenges.

desk verdict A serviceable taxonomy that is let down by unverifiable in-house benchmark numbers and an overclaimed 'first survey' title. read the letter →

arxiv 2412.08291 v1 pith:GROVDYIH submitted 2024-12-11 cs.CL

Nishat Raihan , Christian Newman , Marcos Zampieri This is my paper · ORCID
classification cs.CL
keywords CodeLLMstaxonomygenerationlargelanguagemodelsbenchmarkscorporaencoder-decoderopenproblems
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that Code LLM research has grown too scattered to navigate, and responds with a taxonomy-based survey that classifies the field into five sub-areas: tasks, corpora, models, benchmarks, and challenges. It further sorts coding tasks by input/output type (NL-NL, NL-PL, PL-PL, PL-NL) and sorts models by architecture (encoder-only, encoder-decoder, decoder-only), with decoder-only models split into foundational and fine-tuned categories. The paper compiles performance numbers on code-generation benchmarks to compare decoder and encoder-decoder models, and it closes by listing open problems such as benchmark leakage, multilingual coverage, scaling laws, and synthetic data quality. The intended payoff is that a newcomer can enter the field with a map rather than a pile of papers.

What carries the argument

The central object is the five-branch taxonomy (tasks, corpora, models, benchmarks, challenges), with the task branch as the organizing spine. Tasks are partitioned by the modality of input and output into NL-NL, NL-PL, PL-PL, and PL-NL; Code LLMs are then linked to each branch, so that every model, corpus, benchmark, or challenge in the survey is positioned relative to this scheme. The taxonomy does not itself predict model performance; it carries the argument by making coverage and gaps visible, such as showing that most benchmarks are Python-only and most models are trained on English instructions.

What would settle it

Apply the taxonomy to a random sample of 50 recent Code LLM publications: if a notable share (say, more than one in ten) cannot be assigned to a single leaf of the five-branch scheme without ad-hoc exceptions, the survey's claim of a unified classification system collapses.

Watch

Extended reading notes

Core claim

The central claim is that the entire Code LLM research area can be captured by a single taxonomy, and that organizing the literature this way reveals both the structure of progress and the gaps. The taxonomy's backbone is a task classification based on whether inputs and outputs are natural language or programming language; around that backbone, the survey groups corpora, model architectures, training strategies, benchmarks, and challenges. On the model side, the paper distinguishes encoder-only, encoder-decoder, and decoder-only architectures, and further separates decoder-only models into foundational (base, code, base+code) and fine-tuned variants. On the evaluation side, it treats pass@k on HumanEval and MBPP as the standard measure for generative code models and tabulates reported pass@1 values, including some the authors ran themselves. The open-problems section then extends the taxonomy's logic into a research agenda.

Load-bearing premise

The load-bearing premise is that the pass@1 scores in the main comparison table, including the values the authors say they obtained from their own experiments, were all measured under the same benchmark conditions; the paper provides no experimental setup, prompts, or sampling details to verify that.

Editorial extensions

If this is right

  • Researchers entering the field can locate any given model, dataset, or benchmark within the taxonomy and see which task category it serves.
  • The taxonomy's NL-PL emphasis marks code generation as the central application of decoder-only models, and the paper's pass@1 tables provide a snapshot ranking of the main models as of August 2024.
  • The open-problems section converts the taxonomy into a research agenda, pointing to multilingual benchmarks, blind test sets, quality-focused synthetic data, and a re-examination of scaling laws for code.
  • Because the survey classifies benchmarks by language coverage and test-case counts, it gives readers criteria for choosing an evaluation benchmark for a new code model.
  • The distinction between foundational models (base, code, base+code) and fine-tuned models gives a vocabulary for describing how any new Code LLM was built.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the taxonomy is adopted by the community, it could become a living index: each new Code LLM paper would carry its taxonomy coordinates, making the survey easy to update rather than a point-in-time snapshot.
  • The taxonomy's modality split suggests a natural extension the paper does not develop: multimodal code tasks, such as generating code from screenshots or diagrams, would need a new branch or a redefinition of natural-language input.
  • The authors' own asterisked benchmark numbers could be tested by re-running them on public harnesses; doing so would show how much of the ranking depends on evaluation conditions.
  • The survey's emphasis on synthetic data quality leads to a testable hypothesis: for a fixed compute budget, a small curated programming corpus may outperform a large noisy one, which is a direct experimental follow-up.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper presents a taxonomy-based survey of Code LLMs, organizing the field into tasks, corpora, models, benchmarks, and challenges. It reviews encoder-only, encoder-decoder, and decoder-only architectures; summarizes common pretraining corpora and code-generation benchmarks; and presents a comparative table of pass@1 scores for a wide range of models. The stated contribution is a unified classification framework that can serve as a structured reference for researchers and practitioners entering the Code LLM area, together with a discussion of open problems in benchmarks, multilingual coverage, synthetic data, scaling laws, and resource efficiency.

Significance. If the taxonomy and the associated comparisons were accurate and verifiable, the survey would be a useful entry point for newcomers to the field. The paper's breadth is a genuine strength: it covers encoder-only models, encoder-decoder models, decoder-only base and fine-tuned models, pretraining corpora, benchmarks, and open challenges in one place. The proposed NL-NL/PL-NL/PL-PL/NL-PL task classification is coherent and sensible. However, the survey's central comparative analysis depends on Table IV, and a substantial part of that table consists of unreported experimental numbers from the authors' own runs with no protocol, harness, code, or data release. In a survey, where the value proposition is accuracy and organization, such unverifiable numbers are a load-bearing reliability problem, as are the multiple reference mismatches.

major comments (3)
  1. [Table IV and Section III.C.3] The caption of Table IV states that asterisked entries are "results that are not reported in the original papers and obtained from our own experiments." At least eight entries (phi-1 MBPP and MBPP+, phi-2 MBPP and MBPP+, CodeLLaMA 7B MBPP and MBPP+, StarCoder 7B HumanEval+, SantaCoder MBPP and MBPP+) are presented as original measurements. No evaluation harness, prompt template, sampling temperature, number of samples, or code release is provided. Pass@1 values on EvalPlus-style benchmarks are highly sensitive to these choices, so the numbers cannot be verified and their comparability with the non-asterisked values is asserted rather than established. Because the table is used to rank models and support claims about performance trends, this issue undermines the central comparative analysis. The authors should either remove all own-experiment entries or supply a complete, reproducible evaluation protocol and release the code and data.
  2. [References and citations throughout Section III] Several citations do not match the works they claim to support. HumanEval is cited as [48], which is the GPT-NeoX paper, rather than the OpenAI HumanEval paper; HumanEval+ and MBPP+ are cited as [49], a multilingual programmer paper, rather than the EvalPlus work; GPT3.5 Turbo is cited as [48] in Table IV; Command R+ is cited as [74], which is a paper about reasoning breakdowns rather than the Command R+ model; CommitPack is cited as [70], which is the Chinchilla scaling paper; DBRX is cited as [58], which is the Granite Code Models paper; and Codestral is cited as [53], the Mistral 7B paper. These mismatches make it difficult for readers to trace the survey's factual claims and must be corrected throughout.
  3. [Section III.C.3.a.2] The passage describing code-pretrained models ends mid-sentence with "such as CodeParrot (1TB) [75], which contains about 80". This is not merely a typo; it breaks a substantive claim about a training corpus. The sentence should be completed or removed. The same section also contains a factual inconsistency in Table IV: Claude 3 [Opus] is marked "Only Weights" even though Claude 3 is proprietary and not open-weight, and Codestral is marked "No" despite having a research-license release. These details matter for a survey whose goal is to organize the model landscape accurately.
minor comments (5)
  1. [III.C.3.b] There is a typo "JA V A" that should read "JAVA".
  2. [IV.A.b] The benchmark name is written inconsistently as both "Multiple-E" and "Multipl-E"; the correct name is "Multipl-E".
  3. [III.C.3.b] The model name is written inconsistently as "MagiCoder" in Table IV and "Magicoder" in the text; please unify.
  4. [Table IV] The "Open Source?" column mixes categories such as "Only Weights", "Weights & Data", and "No" without a legend; please define what each term means and use it consistently.
  5. [V] The conclusion states that the survey covers works "rigorously evaluated" on recognized benchmarks, but Table IV includes unreported self-measured values that do not meet this standard; the statement should be reconciled with the table.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: the survey's taxonomy and analysis are organizational; self-citations are minor and non-load-bearing, while Table IV's own-experiment entries raise verifiability issues, not circularity.

full rationale

The paper is a taxonomy-based survey rather than a derivation chain. The proposed taxonomy in Section II—NL-NL, NL-PL, PL-PL, PL-NL tasks, plus corpora, models, benchmarks, and challenges—is a classification scheme; no equation or result in Sections III-V is shown to be equivalent to its inputs by construction, and no fitted parameter is renamed as a prediction. The main self-referential elements are citations to the authors' own benchmarks and papers: mHumanEval [89], CSEPrompts [92,93], Mojo-Coder/MojoBench [86], and the authors' CS-education survey [15]. These are used as literature entries or illustrative resources and are not load-bearing premises that force the taxonomy or the survey's central organizational claim; the survey would not collapse without them. Table IV does contain asterisked pass@1 values described as 'obtained from our own experiments' with no evaluation harness, temperature, sampling, or code, which is a serious reproducibility and reliability flaw and makes those entries unverifiable. However, this is not circularity: the numbers are presented as measurements, not as predictions derived from a fitted model, and the survey's conclusions are not equivalent to those measurements by definition. Additional quality concerns—a truncated sentence in Section III.C.2 ('CodeParrot (1TB) [75], which contains about 80') and mismatched benchmark citations (HumanEval attributed to [48], a GPT-NeoX paper; HumanEval+/MBPP+ attributed to [49])—are correctness issues outside the circularity question. Overall, no significant circularity is present; the low score reflects only the presence of minor, non-load-bearing self-citations.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The survey has no fitted parameters and no invented entities. It relies on two unproven domain assumptions: that pass@1 scores from different papers and the authors' own runs are comparable, and that the five-part taxonomy is a complete organization of the field.

assumptions (2)
  • domain assumption Pass@1 results from different papers and from the authors' own runs are directly comparable.
    Table IV averages pass@1 across models from different sources without accounting for sampling temperature, generation settings, or evaluation harness; this is unstated.
  • domain assumption The taxonomy's five subareas are a complete and faithful organization of Code LLM research.
    Section II presents the taxonomy as a unified classification system, but completeness is asserted, not demonstrated against prior taxonomies.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Code LLMs: A Taxonomy-based Survey." pith.science (2026). https://pith.science/paper/GROVDYIH

@misc{pith2026241208291,
  author       = {Pith},
  title        = {Pith review of: Code LLMs: A Taxonomy-based Survey},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GROVDYIH}},
  note         = {Machine review of arXiv:2412.08291}
}
read the original abstract

Large language models (LLMs) have demonstrated remarkable capabilities across various NLP tasks and have recently expanded their impact to coding tasks, bridging the gap between natural languages (NL) and programming languages (PL). This taxonomy-based survey provides a comprehensive analysis of LLMs in the NL-PL domain, investigating how these models are utilized in coding tasks and examining their methodologies, architectures, and training processes. We propose a taxonomy-based framework that categorizes relevant concepts, providing a unified classification system to facilitate a deeper understanding of this rapidly evolving field. This survey offers insights into the current state and future directions of LLMs in coding tasks, including their applications and limitations.

Figures

Figures reproduced from arXiv: 2412.08291 by the authors.

Figure 1
Figure 1. Code LLMs - Area Taxonomy. • PL-corpora: Datasets composed of programming languages. • NL-corpora: Datasets consisting of natural language text. Recent efforts have focused on compiling specialized pro￾gramming corpora to enhance the models’ understanding of coding tasks. 3) Models: We analyze three primary types of Transformer￾based models [16]: • Encoder-only models: Typically used for under￾standing tasks. • Deco… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

104 extracted references · 23 canonical work pages

  1. [94]

    Pitfalls in Language Models for Code Intelligence: A Taxonomy and Survey

    X. She, Y . Liu, Y . Zhao, Y . He, L. Li, C. Tantithamthavorn, Z. Qin, and H. Wang, “Pitfalls in language models for code intelligence: A taxonomy and survey,” arXiv preprint arXiv:2310.17903 , 2023

    work page Pith review arXiv 2023

  2. [48]

    Gpt-neox-20b: An open- source autoregressive language model,

    S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang et al., “Gpt-neox-20b: An open- source autoregressive language model,” in Proceedings of BigScience Episode# 5–Workshop on Challenges & Perspectives in Creating Large Language Models, 2022, pp. 95–136

    work page 2022

  3. [49]

    Your language model is secretly a multilingual programmer,

    Y . Liu, S. Borgeaud, T. Cai et al., “Your language model is secretly a multilingual programmer,” arXiv preprint arXiv:2303.07636 , 2023

    arXiv 2023

  4. [74]

    Alice in wonderland: Simple tasks showing complete reasoning breakdown in state-of-the-art large language models,

    O. Research, “Alice in wonderland: Simple tasks showing complete reasoning breakdown in state-of-the-art large language models,” arXiv preprint arXiv:2406.02061 , 2024. [Online]. Available: https: //arxiv.org/abs/2406.02061

    arXiv 2024

  5. [70]

    Training compute-optimal large language models,

    J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark et al., “Training compute-optimal large language models,” arXiv preprint arXiv:2203.15556, 2022

    arXiv 2022

  6. [58]

    Granite code models: A family of open foundation models for code intelligence,

    I. A. Research, “Granite code models: A family of open foundation models for code intelligence,” arXiv preprint arXiv:2405.04324 , 2024. [Online]. Available: https://arxiv.org/abs/2405.04324

    arXiv 2024

  7. [75]

    O’Reilly Media, Inc

    L. Tunstall, L. V on Werra, and T. Wolf, Natural language processing with transformers. " O’Reilly Media, Inc.", 2022

    work page 2022

  8. [1]

    Improving language understanding by generative pre-training,

    A. Radford, K. Narasimhan, T. Salimans, I. Sutskever et al., “Improving language understanding by generative pre-training,” 2018

    2018

Show all 104 references
  1. [2]

    Codebert: A pre-trained model for programming and natural languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang et al. , “Codebert: A pre-trained model for programming and natural languages,” in Findings of the Association for Computational Linguistics: EMNLP 2020 , 2020

    2020

  2. [3]

    Graphcodebert: Pre-training code representations with data flow,

    D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu et al. , “Graphcodebert: Pre-training code representations with data flow,” arXiv preprint arXiv:2009.08366 , 2020

    2009 arXiv

  3. [4]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  4. [5]

    Electra: Pre- training text encoders as discriminators rather than generators,

    K. Clark, M.-T. Luong, Q. V . Le, and C. D. Manning, “Electra: Pre- training text encoders as discriminators rather than generators,” arXiv preprint arXiv:2003.10555, 2020

    2003 arXiv

  5. [6]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog, 2019

    2019

  6. [7]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems, 2020

    2020

  7. [8]

    The falcon series of open language models,

    E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojo- caru, M. Debbah, É. Goffinet, D. Hesslow, J. Launay, Q. Malartic et al. , “The falcon series of open language models,” arXiv preprint arXiv:2311.16867, 2023

    2023 arXiv

  8. [9]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  9. [10]

    Roformer: Enhanced transformer with rotary position embedding,

    J. Su, M. Ahmed, Y . Lu, S. Pan, W. Bo, and Y . Liu, “Roformer: Enhanced transformer with rotary position embedding,”Neurocomputing, 2024

    2024

  10. [12]

    Claude technical report,

    Anthropic, “Claude technical report,” https://www.anthropic.com/news/ introducing-claude, 2023

    2023

  11. [13]

    Wikipedia corpus,

    “Wikipedia corpus,” https://dumps.wikimedia.org/, accessed: January 23rd, 2024. [Online]. Available: https://dumps.wikimedia.org/

    2024

  12. [14]

    Common crawl corpus,

    “Common crawl corpus,” https://commoncrawl.org/, accessed: January 23rd, 2024. [Online]. Available: https://commoncrawl.org/

    2024

  13. [15]

    Large language models in computer science education: A systematic literature review,

    N. Raihan, M. L. Siddiq, J. Santos, and M. Zampieri, “Large language models in computer science education: A systematic literature review,” arXiv preprint arXiv:2410.16349 , 2024

    2024 arXiv

  14. [16]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  15. [17]

    On the use of automated text summarization techniques for summarizing source code,

    S. Haiduc, J. Aponte, and A. Marcus, “On the use of automated text summarization techniques for summarizing source code,” in 2010 17th Working Conference on Reverse Engineering . IEEE, 2010, pp. 35–44

    2010

  16. [18]

    Exemplar: A source code search engine for finding highly relevant applications,

    C. McMillan, M. Grechanik, D. Poshyvanyk, Q. Xie, and C. Fu, “Exemplar: A source code search engine for finding highly relevant applications,” IEEE Transactions on Software Engineering , vol. 38, no. 5, pp. 1069–1087, 2011

    2011

  17. [19]

    Using information retrieval based coupling measures for impact analysis,

    D. Poshyvanyk and A. Marcus, “Using information retrieval based coupling measures for impact analysis,” in 2007 IEEE International Conference on Software Maintenance . IEEE, 2007, pp. 29–38

    2007

  18. [20]

    Normalizing source code vocabulary,

    D. Lawrie, D. Binkley, and C. Morrell, “Normalizing source code vocabulary,” in 2010 17th Working Conference on Reverse Engineering . IEEE, 2010, pp. 3–12

    2010

  19. [21]

    Linsen: An efficient approach to split identifiers and expand abbreviations,

    A. Corazza, S. Di Martino, and V . Maggio, “Linsen: An efficient approach to split identifiers and expand abbreviations,” in 2012 28th IEEE International Conference on Software Maintenance (ICSM) . IEEE, 2012, pp. 233–242

    2012

  20. [22]

    Expanding identifiers to normalize source code vocabulary,

    D. Lawrie and D. Binkley, “Expanding identifiers to normalize source code vocabulary,” in 2011 27th IEEE International Conference on Software Maintenance (ICSM) . IEEE, 2011, pp. 113–122

    2011

  21. [23]

    Fowler, Refactoring: Improving the Design of Existing Code

    M. Fowler, Refactoring: Improving the Design of Existing Code . Addison-Wesley Professional, 1999

    1999

  22. [24]

    Detection strategies: Metrics-based rules for detecting design flaws,

    R. Marinescu, “Detection strategies: Metrics-based rules for detecting design flaws,” in Proceedings of the 20th IEEE International Conference on Software Maintenance . IEEE, 2004, pp. 350–359

    2004

  23. [25]

    Gpt-4 technical report,

    OpenAI, “Gpt-4 technical report,” https://arxiv.org/abs/2303.08774, 2023

    2023 arXiv

  24. [26]

    Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,

    Y . Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , 2021

    2021

  25. [27]

    Code llama: Open foundation models for code,

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, T. Remez, J. Rapin et al., “Code llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950 , 2023

    2023 arXiv

  26. [28]

    Ernie: Enhanced language representation with informative entities,

    Z. Zhang, X. Han, Z. Liu, X. Jiang, M. Sun, and Q. Liu, “Ernie: Enhanced language representation with informative entities,” in Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    2019

  27. [29]

    A robustly optimized bert pre- training approach with post-training,

    Z. Liu, W. Lin, Y . Shi, and J. Zhao, “A robustly optimized bert pre- training approach with post-training,” in China National Conference on Chinese Computational Linguistics . Springer, 2021

    2021

  28. [30]

    Learning and evaluating contextual embedding of source code,

    A. Kanade, P. Maniatis, G. Balakrishnan, and K. Shi, “Learning and evaluating contextual embedding of source code,” in International conference on machine learning . PMLR, 2020

    2020

  29. [31]

    Treebert: A tree- based pre-trained model for programming language,

    X. Jiang, Z. Zheng, C. Lyu, L. Li, and L. Lyu, “Treebert: A tree- based pre-trained model for programming language,” in Uncertainty in Artificial Intelligence. PMLR, 2021

    2021

  30. [32]

    Syncobert: Syntax-guided multi-modal contrastive pre-training for code representation,

    X. Wang, Y . Wang, F. Mi, P. Zhou, Y . Wan, X. Liu, L. Li, H. Wu, J. Liu, and X. Jiang, “Syncobert: Syntax-guided multi-modal contrastive pre-training for code representation,” arXiv preprint arXiv:2108.04556 , 2021

    2021 arXiv

  31. [33]

    Code-mvp: Learning to represent source code from multiple views with contrastive pre-training,

    X. Wang, Y . Wang, Y . Wan, J. Wang, P. Zhou, L. Li, H. Wu, and J. Liu, “Code-mvp: Learning to represent source code from multiple views with contrastive pre-training,” in Findings of the Association for Computational Linguistics: NAACL 2022 , 2022

    2022

  32. [34]

    Codesearchnet challenge: Evaluating the state of semantic code search,

    H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “Codesearchnet challenge: Evaluating the state of semantic code search,” arXiv preprint arXiv:1909.09436 , 2019

    1909 arXiv

  33. [35]

    Summarizing source code using a neural attention model,

    S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer, “Summarizing source code using a neural attention model,” in 54th Annual Meeting of the Association for Computational Linguistics 2016 . Association for Computational Linguistics, 2016

    2016

  34. [36]

    Python 150k: A large-scale dataset for python programs,

    E. Z. SRI Lab, “Python 150k: A large-scale dataset for python programs,” https://www.sri.inf.ethz.ch/py150, accessed: January 26, 2024

    2024

  35. [37]

    Cosqa: 20,000+ web queries for code search and question answering,

    J. Huang, D. Tang, L. Shou, M. Gong, K. Xu, D. Jiang, M. Zhou, and N. Duan, “Cosqa: 20,000+ web queries for code search and question answering,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conferen...

    2021

  36. [38]

    Towards a big data curated benchmark of inter-project code clones,

    J. Svajlenko, J. F. Islam, I. Keivanloo, C. K. Roy, and M. M. Mia, “Towards a big data curated benchmark of inter-project code clones,” in 2014 IEEE International Conference on Software Maintenance and Evolution. IEEE, 2014

    2014

  37. [39]

    Global relational models of source code,

    V . J. Hellendoorn, C. Sutton, R. Singh, P. Maniatis, and D. Bieber, “Global relational models of source code,” in International conference on learning representations , 2019

    2019

  38. [40]

    Scalpel: The python static analysis framework,

    L. Li, J. Wang, and H. Quan, “Scalpel: The python static analysis framework,” arXiv preprint arXiv:2202.11840 , 2022

    2022 arXiv

  39. [41]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,”The Journal of Machine Learning Research, 2020

    2020

  40. [42]

    Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and compre- hension,

    M. Lewis, Y . Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V . Stoyanov, and L. Zettlemoyer, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and compre- hension,” in Proceedings of the 58th Annual Meeting of the Associa...

    2020

  41. [43]

    Pymt5: multi-mode translation of natural language and python code with transformers,

    C. Clement, D. Drain, J. Timcheck, A. Svyatkovskiy, and N. Sundaresan, “Pymt5: multi-mode translation of natural language and python code with transformers,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2020

    2020

  42. [44]

    Codet5+: Open code large language models for code understanding and generation,

    Y . Wang, H. Le, A. D. Gotmare, N. D. Bui, J. Li, and S. C. Hoi, “Codet5+: Open code large language models for code understanding and generation,” arXiv preprint arXiv:2305.07922 , 2023

    2023 arXiv

  43. [45]

    Spt-code: Sequence- to-sequence pre-training for learning source code representations,

    C. Niu, C. Li, V . Ng, J. Ge, L. Huang, and B. Luo, “Spt-code: Sequence- to-sequence pre-training for learning source code representations,” in Proceedings of the 44th International Conference on Software Engineering, 2022

    2022

  44. [46]

    Competition-level code generation with alphacode,

    Y . Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago et al., “Competition-level code generation with alphacode,” Science, 2022

    2022

  45. [47]

    Unified pre- training for program understanding and generation,

    W. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “Unified pre- training for program understanding and generation,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , 2021

    2021

  46. [50]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le et al., “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732 , 2021

    2021 arXiv

  47. [51]

    Introducing the next generation of Claude,

    Anthropic, “Introducing the next generation of Claude,” https://www. anthropic.com/news/claude-3-family, Mar. 2024, accessed: 2024-03-06

    2024

  48. [52]

    Gpt-4 omni: A comprehensive multimodal model for language, vision, and beyond,

    OpenAI, “Gpt-4 omni: A comprehensive multimodal model for language, vision, and beyond,” arXiv preprint arXiv:2408.01234 , 2024. [Online]. Available: https://arxiv.org/abs/2408.01234

    2024 arXiv

  49. [55]

    The llama 3 herd of models,

    M. A. Research, “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024. [Online]. Available: https://arxiv.org/abs/2407. 21783

    2024 arXiv

  50. [56]

    Wizardcoder: Empowering code large language models with evol-instruct,

    Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, and D. Jiang, “Wizardcoder: Empowering code large language models with evol-instruct,” arXiv preprint arXiv:2306.08568 , 2023

    2023 arXiv

  51. [57]

    Nemotron-4 340b: Technical report on large-scale code models,

    N. A. Research, “Nemotron-4 340b: Technical report on large-scale code models,” arXiv preprint arXiv:2406.11704 , 2024. [Online]. Available: https://arxiv.org/abs/2406.11704

    2024 arXiv

  52. [59]

    Magicoder: Source code is all you need,

    Y . Wei, Z. Wang, J. Liu, Y . Ding, and L. Zhang, “Magicoder: Source code is all you need,” arXiv preprint arXiv:2312.02120 , 2023

    2023 arXiv

  53. [60]

    phind-codellama,

    phind, “phind-codellama,” https://www.phind.com/blog/ code-llama-beats-gpt4, 2023, accessed: January 28. 2024

    2023

  54. [61]

    Gemini: a family of highly capable multimodal models,

    G. Team, R. Anil, S. Borgeaud, Y . Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth et al., “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805 , 2023

    2023 arXiv

  55. [62]

    Textbooks are all you need,

    S. Gunasekar, Y . Zhang, J. Aneja, C. C. T. Mendes, A. Del Giorno, S. Gopi, M. Javaheripi, P. Kauffmann, G. de Rosa, O. Saarikivi et al., “Textbooks are all you need,” arXiv preprint arXiv:2306.11644 , 2023

    2023 arXiv

  56. [63]

    Textbooks are all you need ii: phi-1.5 technical report,

    Y . Li, S. Bubeck, R. Eldan, A. Del Giorno, S. Gunasekar, and Y . T. Lee, “Textbooks are all you need ii: phi-1.5 technical report,” arXiv preprint arXiv:2309.05463, 2023

    2023 arXiv

  57. [64]

    Starcoder 2 and the stack v2: The next generation,

    A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y . Wei et al., “Starcoder 2 and the stack v2: The next generation,” arXiv preprint arXiv:2402.19173 , 2024

    2024 arXiv

  58. [65]

    Palm 2 technical report,

    R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen et al. , “Palm 2 technical report,” arXiv preprint arXiv:2305.10403 , 2023

    2023 arXiv

  59. [66]

    Starcoder: may the source be with you!

    R. Li, L. B. Allal, Y . Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim et al. , “Starcoder: may the source be with you!” arXiv preprint arXiv:2305.06161 , 2023. [Online]. Available: https://arxiv.org/pdf/2305.06161.pdf

    2023 arXiv

  60. [67]

    Mistral 7b,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier et al., “Mistral 7b,” arXiv preprint arXiv:2310.06825 , 2023

    2023 arXiv

  61. [68]

    Santacoder: don’t reach for the stars!

    L. B. Allal, R. Li, D. Kocetkov, C. Mou, C. Akiki, C. M. Ferrandis, N. Muennighoff, M. Mishra, A. Gu, M. Dey et al., “Santacoder: don’t reach for the stars!” arXiv preprint arXiv:2301.03988 , 2023

    2023 arXiv

  62. [69]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  63. [71]

    Roformer: En- hanced transformer with rotary position embedding,

    J. Su, Y . Lu, S. Pan, B. Wen, and Y . Liu, “Roformer: En- hanced transformer with rotary position embedding,” arXiv preprint arXiv:2104.09864, 2021

    2021 arXiv

  64. [72]

    Self-attention with relative position representations,

    P. Shaw, J. Uszkoreit, and A. Vaswani, “Self-attention with relative position representations,” arXiv preprint arXiv:1803.02155 , 2018

    2018 arXiv

  65. [73]

    Mixtral of experts,

    A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand et al., “Mixtral of experts,” arXiv preprint arXiv:2401.04088 , 2024

    2024 arXiv

  66. [76]

    Glu variants improve transformer,

    N. Shazeer, “Glu variants improve transformer,” arXiv preprint arXiv:2002.05202, 2020

    2002 arXiv

  67. [77]

    Searching for activation functions,

    P. Ramachandran, B. Zoph, and Q. V . Le, “Searching for activation functions,” in arXiv preprint arXiv:1710.05941 , 2017

    2017 arXiv

  68. [78]

    Language modeling with gated convolutional networks,

    Y . N. Dauphin, A. Fan, M. Auli, and D. Grangier, “Language modeling with gated convolutional networks,” in Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017

    2017

  69. [79]

    Rectified linear units improve restricted boltzmann machines,

    V . Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” Proceedings of the 27th international conference on machine learning (ICML-10) , 2010

    2010

  70. [80]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685 , 2021

    2021 arXiv

  71. [81]

    Qlora: Efficient finetuning of quantized llms,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” arXiv preprint arXiv:2305.14314, 2023

    2023 arXiv

  72. [82]

    Wizardlm: Empowering large language models to follow complex instructions,

    C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, and D. Jiang, “Wizardlm: Empowering large language models to follow complex instructions,” arXiv preprint arXiv:2304.12244 , 2023

    2023 arXiv

  73. [83]

    Octopack: Instruction tuning code large language models,

    N. Muennighoff, Q. Liu, A. Zebaze, Q. Zheng, B. Hui, T. Y . Zhuo, S. Singh, X. Tang, L. V on Werra, and S. Longpre, “Octopack: Instruction tuning code large language models,” arXiv preprint arXiv:2308.07124 , 2023

    2023 arXiv

  74. [84]

    Codegemma: Open code language models for code generation and analysis,

    C. Yadav, L. Raphaël, A. Glaese, M. Allamanis, M. Alberti, M. Bošnjak, V . Botev, X. Chen, F. Fredrikson, A. L. Gaunt et al. , “Codegemma: Open code language models for code generation and analysis,” arXiv preprint arXiv:2403.05523, 2024

    2024 arXiv

  75. [85]

    Gemma: Open models based on gemini research and technol- ogy,

    Google, “Gemma: Open models based on gemini research and technol- ogy,” https://blog.google/technology/developers/gemma-open-models/, 2024, accessed: 2024-03-14

    2024

  76. [86]

    Mojobench: Language modeling and benchmarks for mojo,

    N. Raihan, J. Santos, and M. Zampieri, “Mojobench: Language modeling and benchmarks for mojo,” arXiv preprint arXiv:2410.17736 , 2024

    2024 arXiv

  77. [87]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021

    2021 arXiv

  78. [88]

    Humaneval-xl: A multilingual code generation benchmark for cross-lingual natural language generalization,

    Q. Peng, Y . Chai, and X. Li, “Humaneval-xl: A multilingual code generation benchmark for cross-lingual natural language generalization,” arXiv preprint arXiv:2402.16694 , 2024

    2024 arXiv

  79. [89]

    mhumaneval–a multilingual benchmark to evaluate large language models for code generation,

    N. Raihan, A. Anastasopoulos, and M. Zampieri, “mhumaneval–a multilingual benchmark to evaluate large language models for code generation,” arXiv preprint arXiv:2410.15037 , 2024

    2024 arXiv

  80. [90]

    Multipl-e: A scalable and extensible approach to benchmarking neural code generation,

    F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M.-H. Yee, Y . Zi, C. J. Anderson, M. Q. Feldman et al., “Multipl-e: A scalable and extensible approach to benchmarking neural code generation,” arXiv preprint arXiv:2208.08227 , 2022

    2022 arXiv

  81. [91]

    Ds-1000: A natural and reliable benchmark for data science code generation,

    Y . Lai, C. Li, Y . Wang, T. Zhang, R. Zhong, L. Zettlemoyer, W.- t. Yih, D. Fried, S. Wang, and T. Yu, “Ds-1000: A natural and reliable benchmark for data science code generation,” in International Conference on Machine Learning . PMLR, 2023

    2023

  82. [92]

    Cseprompts: A benchmark of introductory computer science prompts,

    N. Raihan, D. Goswami, S. S. C. Puspo, C. Newman, T. Ranasinghe, and M. Zampieri, “Cseprompts: A benchmark of introductory computer science prompts,” in International Symposium on Methodologies for Intelligent Systems. Springer, 2024, pp. 45–54

    2024

  83. [93]

    On the performance of large language models on introductory programming assignments,

    N. Raihan, D. Goswami, S. S. C. Puspo, M. L. Siddiq, C. Newman, T. Ranasinghe, J. C. Santos, and M. Zampieri, “On the performance of large language models on introductory programming assignments,” 2024

    2024

  84. [95]

    Holistic evaluation of language models,

    P. Liang et al., “Holistic evaluation of language models,” arXiv preprint arXiv:2211.09110, 2022

    2022 arXiv

  85. [96]

    Codexglue: A machine learning benchmark dataset for code understanding and generation,

    S. Lu, D. Guo, S. Ren et al. , “Codexglue: A machine learning benchmark dataset for code understanding and generation,” arXiv preprint arXiv:2102.04664, 2021

    2021 arXiv

  86. [97]

    An impact study of code quality and quantity on ai code generation,

    H. Peng, T. W. Chen, K. Goel et al., “An impact study of code quality and quantity on ai code generation,” arXiv preprint arXiv:2301.10226 , 2023

    2023 arXiv

  87. [98]

    Software robustness for ai/ml components,

    F. Pierazzi, S. Guarnieri, and M. Costa, “Software robustness for ai/ml components,” arXiv preprint arXiv:2009.10730 , 2020

    2009 arXiv

  88. [99]

    Asleep at the keyboard? assessing the security of github copilot’s code contributions,

    H. Pearce, B. Ahmad, B. Tan et al., “Asleep at the keyboard? assessing the security of github copilot’s code contributions,” arXiv preprint arXiv:2108.09293, 2022

    2022 arXiv

  89. [100]

    The unsolved problems in ai security,

    D. Hendrycks, A. Zou, and D. Song, “The unsolved problems in ai security,” arXiv preprint arXiv:2109.13975 , 2021

    2021 arXiv

  90. [101]

    The stack: 3 tb of permissively licensed source code,

    D. Kocetkov, R. Li, L. B. Allal, J. Li, C. Mou, C. M. Ferrandis, Y . Jernite, M. Mitchell, S. Hughes, T. Wolf et al., “The stack: 3 tb of permissively licensed source code,” arXiv preprint arXiv:2211.15533 , 2022

    2022 arXiv

  91. [102]

    Bloom: A 176b-parameter open-access multilingual language model,

    B. Workshop, T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ili ´c, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon et al. , “Bloom: A 176b-parameter open-access multilingual language model,” arXiv preprint arXiv:2211.05100, 2022

    2022 arXiv

  92. [103]

    Aya model: An instruction finetuned open-access multilingual language model,

    A. Üstün, V . Aryabumi, Z.-X. Yong, W.-Y . Ko, D. D’souza, G. Onilude, N. Bhandari, S. Singh, H.-L. Ooi, A. Kayid et al. , “Aya model: An instruction finetuned open-access multilingual language model,” arXiv preprint arXiv:2402.07827, 2024

    2024 arXiv

  93. [104]

    Better to ask in english: Cross-lingual evaluation of large language models for healthcare queries,

    Y . Jin, M. Chandra, G. Verma, Y . Hu, M. De Choudhury, and S. Kumar, “Better to ask in english: Cross-lingual evaluation of large language models for healthcare queries,” arXiv e-prints, 2023

    2023

  94. [105]

    The era of 1-bit llms: All large language models are in 1.58 bits,

    S. Ma, H. Wang, L. Ma, L. Wang, W. Wang, S. Huang, L. Dong, R. Wang, J. Xue, and F. Wei, “The era of 1-bit llms: All large language models are in 1.58 bits,” arXiv preprint arXiv:2402.17764 , 2024

    2024 arXiv

  95. [106]

    Unsupervised cross-lingual representation learning at scale,

    A. Conneau, K. Khandelwal, N. Goyal, V . Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V . Stoyanov, “Unsupervised cross-lingual representation learning at scale,” arXiv preprint arXiv:1911.02116, 2019

    1911 arXiv

  96. [107]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,

    J. Liu, C. S. Xia, Y . Wang, and L. Zhang, “Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.