Pith. sign in

REVIEW 3 major objections 5 minor 40 references

LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generation

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Public library evolution measurably degrades code completion models.

desk verdict Useful benchmark for version-sensitive code completion, but the RAG mitigation claim is undermined by oracle-derived retrieval queries. read the letter →

arxiv 2412.04478 v1 pith:U5ENW2WJ submitted 2024-11-19 cs.SE cs.AI

Sachit Kuhar , Wasi Uddin Ahmad , Zijian Wang , Nihal Jain , Haifeng Qian , Baishakhi Ray , Murali Krishna Ramanathan , Xiaofei Ma
show 1 more author
Anoop Deoras
This is my paper · ORCID
classification cs.SEcs.AI
keywords codecompletionlibraryevolutionAPIdeprecationversion-specificdocumentationretrieval-augmentedgenerationbenchmarkembeddingmodelsknowledgecutoff
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LibEvolutionEval tests whether code completion models can keep up as public libraries change their APIs. The paper argues that model performance tracks library versions: scores fluctuate as torch, matplotlib, and six other libraries release new versions, and models systematically struggle with deprecated APIs. Feeding models retrieved documentation for the exact library version improves completion F1, but a version-dependent bias remains. The result matters because real developers work across old and new library versions, so a coding assistant that is right for one version can be wrong for another.

What carries the argument

The load-bearing object is the version-specific code-completion benchmark itself: a dataset of in-line completions, each anchored to a documented library version and annotated by API relationship (direct versus indirect) and by granularity (introduced, deprecated, modified, unchanged). Direct completions are import-prefixed calls such as torch.nn.ReLU(); indirect completions resolve a method on an object created by an earlier API call. The paper also builds version-stamped documentation databases for PyTorch and Matplotlib and uses an embedding model to retrieve the top three matching API docs, which are injected as commented context. The F1 score over extracted API expressions carries the evaluation, and Mean Reciprocal Rank carries the retrieval analysis.

What would settle it

Run the same version-specific completions on snippets that have been paraphrased or had identifiers renamed so they no longer match any training example, then compare F1; if the version-dependent gaps disappear under paraphrase, the measured evolution bias is largely memorization, while persistence would support the paper's interpretation.

Watch

Extended reading notes

Core claim

The paper's central claim is that public library evolution, not just code context, drives code completion accuracy. Across eight libraries and several open models, the authors show F1 varies substantially from version to version, with deprecated APIs consistently harder than the overall set and newly introduced APIs exposing a model's training cutoff. They introduce a benchmark with two complementary settings---realistic snippets from permissively licensed repositories and controlled templates built from API documentation---and show that adding retrieved version-specific documentation improves performance (for example, StarCoder2-7B rises from 68.8 to 73.3 F1 on PyTorch) without eliminating the underlying version bias. The conclusion is that developers using code assistants with different library versions experience different accuracy, and fixes like retrieval are partial.

Load-bearing premise

The result depends on the benchmark's version-sensitive scores actually measuring API-evolution understanding, which requires that the evaluated models did not memorize the exact GitHub snippets used as evaluation examples; the paper's own limitations section concedes that code LMs are trained on vast unlabeled code repositories and may have seen this data.

Editorial extensions

If this is right

  • Code completion benchmarks that ignore library version overstate or hide model capability, so version-controlled evaluation is needed.
  • Version-aware retrieval should become a standard component in coding assistants, since it consistently improves completion F1.
  • Deprecated-API performance can serve as a probe of a model's temporal knowledge; for instance, CodeGen-1.0's drop on 2023-2024 introduced APIs aligns with its 2022 knowledge cutoff.
  • Scaling models improves overall performance but does not remove version-to-version fluctuations, pointing to a need for versioned fine-tuning or explicit temporal training signals.
  • Embedding models used for retrieval also drift with library versions, which explains why retrieval augmentation cannot fully close the version gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The benchmark could be reused as a contamination test: if a model scores high on realistic GitHub examples but low on paraphrased or identifier-renamed versions of the same API calls, memorization rather than version understanding is at play.
  • Version-stamped documentation retrieval mitigates surface errors but will not fix internal version priors, suggesting that explicit version tokens in prompts or calibrated confidence scores are worth exploring.
  • The direct-versus-indirect gap hints that models track object provenance poorly, so a testable extension is whether chain-of-thought reasoning about the originating API improves indirect completions.
  • Library-evolution benchmarks could be extended beyond Python to other ecosystems to measure how widely version bias affects assistant reliability and legacy-code maintenance.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces LibEvolutionEval, a benchmark for version-specific code completion covering eight Python libraries across multiple versions, with two evaluation scenarios: realistic GitHub-derived code and controlled documentation-derived templates for PyTorch and Matplotlib. It evaluates several code LLMs (Mistral, StarCoder2, GPT-4o-mini, CodeGen) under in-file, version-aware, and version-aware-RAG contexts, and embedding models on version-specific documentation retrieval. The main empirical claims are that model performance varies with library version, that version-specific retrieved documentation improves F1 but does not eliminate version bias, that indirect API completions are easier than direct ones, and that deprecated and newly introduced APIs induce drops (including knowledge-cutoff effects). The paper concludes that code assistants are version-sensitive and that retrieval partially mitigates the issue.

Significance. LibEvolutionEval fills a real gap: existing code-completion benchmarks mostly ignore library version evolution, while this benchmark explicitly ties code-completion examples to library versions and labels APIs by introduced/deprecated/modified status. The controlled synthetic setting and the knowledge-cutoff analysis (Table 4b) are particularly valuable, as they offer a falsifiable check of model temporal awareness. If the methodological issues are fixed, the benchmark would be a solid resource for studying version sensitivity of code models and for evaluating retrieval-augmented completion. The study also provides a useful negative result: even with version-aware context, large performance variance across versions remains. The paper is not suitable for acceptance in its current form because the RAG mitigation results are uninterpretable due to query construction leakage.

major comments (3)
  1. [Section 2.3, Appendix C] The retrieval query used in the '+Version-Aware RAG' condition is generated by giving the target code completion to Claude v2 and asking it to produce a natural-language instruction without revealing the API name or arguments. This makes the query oracle-conditioned: at inference time a developer would only have the prefix, not the gold completion. Consequently, the Table 3 gains (e.g., +4.5/+5.7 F1 for StarCoder2 and +5.4/+14.2 for GPT-4o-mini) conflate the effect of version-specific documentation with a leakage of the answer into the retrieval step. The second half of the central claim, that providing version-specific documentation improves completion, is therefore not supported by the reported experiments. Please rerun the RAG condition with queries generated from the left context only (e.g., from the code prefix), or at least report an ablation in which the query is derived from the prefix and show that the gains persist.
  2. [Appendix A] The paper acknowledges that CodeLMs may have encountered the evaluation data during pretraining, but it does not measure the degree of overlap. Because the main version-sensitivity results in the realistic scenario (Figure 5 and Table 3) are computed on GitHub-derived snippets, the observed version-dependent F1 gaps could be driven by the distribution of those snippets in the training corpus rather than by the model's inability to adapt to API evolution. Please quantify contamination (e.g., n-gram overlap with StarCoder/Stack, or report results for the controlled documentation-derived examples separately and show that the version-sensitivity pattern persists) and discuss how the conclusions change under deduplication.
  3. [Section 4, Table 5] The eight-library overview includes very small per-version samples: pyyaml has 8 examples per year, tqdm 12, and pillow 40. The version-by-version F1 fluctuations reported in Figure 5 for these libraries are therefore likely to be dominated by sampling noise, yet the text states that 'developer experience can vary significantly across all models and libraries.' Please report confidence intervals or otherwise restrict the breadth claim to libraries with adequate per-version sample sizes, or pool the small libraries.
minor comments (5)
  1. [Abstract, Section 1] The phrase 'as they evolve over the year' is ambiguous; the data cover versions from 2019 to 2024. Use 'over multiple years' for consistency.
  2. [General] The manuscript does not state where the benchmark and preprocessing code will be released. Add a data and code availability statement.
  3. [Table 4c] The column header 'Deprecated/Intro-duced' is a line-break artifact; please use 'Deprecated/Introduced' in the table.
  4. [Section 3, Models] The three models employ different completion strategies (FIM, left-context-only, instruction-based), so absolute F1 scores across models are not directly comparable. State explicitly that only within-model comparisons (In-File vs Version-Aware vs RAG) are meaningful.
  5. [Figure 6] The subfigures (a)-(f) are referenced in an order that does not match their caption order (e.g., Figure 6b is referenced before Figure 6a). Please renumber or reference the subfigures consistently.

Circularity Check

1 steps flagged · score 6.0 of 10

The version-aware RAG mitigation claim is undermined by oracle-conditioned queries: the retrieval query is generated from the target code completion, so the +RAG gains in Table 3 reduce by construction to answer leakage.

  1. self definitional [Section 2.3, 'Version-Specific Retrieved API-Context'; Appendix C, Figure 10.]
    "For each code completion task, we generate a query using the natural language instruction describing the developer’s intent. This is done by giving target code completion to Anthropic’s Claude v2 (Anthropic, 2023) and asking it to not reveal the regular expressions corresponding to the name and input arguments of target code completion (refer to § C for the Appendix and Figure 2-(H))."

    The retrieval query is constructed from the target code completion, i.e., from the ground-truth answer the model is supposed to predict. The top-3 retrieved API documents are then inserted into the prompt before the left context, so the model receives context derived from the exact API expression being evaluated. In real deployment, a developer only has the prefix, not the reference completion. The reported Version-Aware RAG improvements over In-File in Table 3 (e.g., StarCoder2: 68.8 to 73.3 on PyTorch; GPT-4o-mini: 52.5 to 66.7 on Matplotlib) are therefore uninterpretable as evidence that version-specific documentation retrieval helps: the query already encodes information about the target completion, making the retrieval step oracle-conditioned.

full rationale

The paper contains no derivation chain in the usual sense, but its central mitigation claim is partially circular. The first claim—that public library evolution significantly influences code completion performance—is supported by independent evidence that does not rely on the flawed query construction: version-by-version F1 variation in Figure 5, deprecated-API underperformance in Table 4a and 4c, and the CodeGen knowledge-cutoff effect on introduced APIs in Table 4b. Those analyses are self-contained and do not reduce to their inputs. The circular step is confined to the '+Version-Aware RAG' rows of Table 3 and the retrieval evaluation: the query is generated by giving Claude the target code completion, so retrieval is conditioned on the answer. This makes the claimed RAG gains a function of the oracle-derived query rather than a deployable retrieval-augmented workflow. Appendix A also candidly notes possible train/evaluation overlap, which is a validity concern but not a circularity. Overall score 6: one central 'prediction' reduces by construction, while other benchmark contributions retain independent content.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The benchmark rests on several practical assumptions about version labeling, metric validity, and absence of memorization. No physical or theoretical entities are introduced. The main methodological choices (top-k, context lengths, subset sizes) are hand-picked rather than fitted.

free parameters (4)
  • RAG retrieval top-k = 3
    Section 2.3 fixes the number of retrieved API docs to 3 without sensitivity analysis; the RAG results in Table 3 depend on this choice.
  • context window trim = 4096 tokens
    The inference section trims contexts to the nearest 4K tokens; results may vary with context length.
  • max generation length = 128 tokens
    The inference section sets a cap of 128 generated tokens; no ablation is reported.
  • deprecated/introduced subset size = 40 per version
    Tables 4a to 4c and appendix F show focused subsets of 40 examples per version, a hand-chosen sample size that makes per-version F1 very noisy.
assumptions (5)
  • domain assumption The library version stated in each repository's requirements.txt, or in the dependency matrix, is the version the code actually runs against.
    Section 2.1 bases all realistic-scenario version labels on requirements.txt and PyPI dependency matrices; inaccurate pins would mislabel examples and confound version effects.
  • domain assumption The evaluation snippets were not memorized by the evaluated LLMs during pre-training.
    The paper states in Appendix A that CodeLMs are trained on large unlabeled code corpora and may have seen the evaluation data; the study assumes this overlap does not drive the results.
  • domain assumption The API name/signature F1 score is a valid measure of completion correctness.
    Section 3 defines the metric as F1 on extracted API expressions; functionally equivalent completions that differ syntactically are counted wrong.
  • ad hoc to paper The synthetic template examples from API documentation are representative of developer code.
    Section 2.1 constructs controlled examples by converting documentation into a fixed template (Figure 8); this narrow distribution may not generalize.
  • ad hoc to paper Claude v2-generated queries from gold completions approximate developer intent without leaking the answer.
    Section 2.3 builds retrieval queries by prompting Claude with the target completion and instructing it to hide names and arguments; this is an unverified proxy and leaks answer structure.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generation." pith.science (2026). https://pith.science/paper/U5ENW2WJ

@misc{pith2026241204478,
  author       = {Pith},
  title        = {Pith review of: LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U5ENW2WJ}},
  note         = {Machine review of arXiv:2412.04478}
}
read the original abstract

Recent advancements in code completion models have primarily focused on local file contexts. However, these studies do not fully capture the complexity of real-world software development, which often requires the use of rapidly-evolving public libraries. To fill the gap, we introduce LibEvolutionEval, a detailed study requiring an understanding of library evolution to perform in-line code completion accurately. LibEvolutionEval provides a version-specific code-completion task comprised of eight libraries (torch, torchvision, scipy, pil, tqdm, pyyaml, matplotlib, and pandas) as they evolve over the year along with a detailed analysis of the evolution of two popular and well-maintained public libraries: PyTorch and Matplotlib. We evaluate popular public models and find that public library evolution significantly influences model performance. We explored mitigation methods by studying how retrieved version-specific library documentation and prompting can improve the model's capability in handling these fast-evolving packages, paving a promising future path in better handling fast-evolving libraries.

Figures

Figures reproduced from arXiv: 2412.04478 by the authors.

Figure 1
Figure 1. An example of a code completion scenario under [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. LIBEVOLUTIONEVAL’s preprocessing pipeline to obtain version-specific code-completions meta-data including documentation, NL instructions, and API annotation. 2 LIBEVOLUTIONEVAL: Version-Specific Code Completions Each code completion example in LIBEVOLU￾TIONEVAL consists of code prompts ending at a position where the LLM is tasked to complete the missing expression, typically involving one or more API calls to the pu… view at source ↗
Figure 3
Figure 3. APIs classification based on completion type. same source (e.g. GitHub repository). This means that if we have already included an example of a specific API (e.g., torch.ones) from a particular source, we will exclude any additional examples from that same source that use the same API (i.e., torch.ones). This strategy aims to ensure the di￾versity of API calls within the evaluation dataset. Documentation Collection … view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Illustration of the evolution of PyTorch and Matplotlib public libraries over time. This highlights the rapid [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Illustration of the code completion performance of the Starcoder2, Mistral, and GPT-4o-mini models by [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Detailed analysis of the impact on code completion as [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 9
Figure 9. Figure 9: Qualitative example of an error in code com [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 7
Figure 7. Figure 7: Examples of API evaluation [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Matplotlib API evaluation example created [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 12
Figure 12. Figure 12: Larger Starcoder and Starcoder2 models perform better at code completions. F Detailed Statistics of Data Main Result Library 2019 2020 2021 2022 2023 torch 286 513 708 835 738 torchvision 16 16 14 16 16 scipy 70 70 70 70 68 pandas 60 60 60 59 60 matplotlib 90 90 90 89…
Figure 11
Figure 11. Figure 11: Larger CodeSage models perform better at [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 13
Figure 13. Figure 13: Visualizing prompts given to code LLMs in [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 16 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Anthropic. 2023. https://www.anthropic.com/news/claude-2 Claude

    work page 2023

  4. [4]

    Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ramanathan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Sudi...

    work page 2023

  5. [5]

    Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. 2022. https://arxiv.org/abs/2207.14255 Efficient training of language models to fill in the middle . arXiv preprint arXiv:2207.14255

    arXiv 2022

  6. [6]

    Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. 2023. https://doi.org/10.1109/TSE.2023.3267446 Multipl-e: A scalable and polyglot approach to benchmarking neural code generation . IEEE Trans...

    arXiv 2023

  7. [7]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. https://arxiv.org/abs/2107.03374 Evaluating large language models trained on code . ArXiv preprint, abs/2107.03374

    arXiv 2021

  8. [8]

    Colin Clement, Shuai Lu, Xiaoyu Liu, Michele Tufano, Dawn Drain, Nan Duan, Neel Sundaresan, and Alexey Svyatkovskiy. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.387 Long-range modeling of source code files with e WASH : Extended window access by syntax hierarchy . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process...

    work page doi:10.18653/v1/2021.emnlp-main.387 2021

Show all 40 references
  1. [9]

    Hantian Ding, Varun Kumar, Yuchen Tian, Zijian Wang, Rob Kwiatkowski, Xiaopeng Li, Murali Krishna Ramanathan, Baishakhi Ray, Parminder Bhatia, and Sudipta Sengupta. 2023 a . https://doi.org/10.18653/v1/2023.acl-industry.34 A static evaluation of code completion by large langua...

    2023 doi

  2. [10]

    Ahmad, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang

    Yangruibo Ding, Zijian Wang, Wasi U. Ahmad, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2024. https://aclanthology.org/2024.lrec-main.305 C o C o MIC : Code completion by jointly modeling in-file and cross-file context . In Proceedi...

    2024

  3. [11]

    Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2023 b . https://openreview.net/forum?id=wgDcbBMSfh Crosscodeeval: A diverse and multilingual benchmark f...

    2023

  4. [12]

    Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. Unixcoder: Unified cross-modal pre-training for code representation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7212--7225

    2022

  5. [13]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. https://openreview.net/forum?id=sD93GOzH3i5 Measuring coding challenge competence with APPS . In Thirty-fifth Co...

    2021

  6. [14]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  7. [15]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. https://openreview.net/forum?id=VTF8yNQM66 SWE -bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Re...

    2024

  8. [16]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. https://arxiv.org/abs/2305.06161 Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161

    2023 arXiv

  9. [17]

    Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R \'e mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal,...

    2022 doi

  10. [18]

    Dianshu Liao, Shidong Pan, Qing Huang, Xiaoxue Ren, Zhenchang Xing, Huan Jin, and Qinying Li. 2023. Context-aware code generation framework for code repositories: Local, global, and third-party library awareness. arXiv preprint arXiv:2312.05772

    2023 arXiv

  11. [19]

    Tianyang Liu, Canwen Xu, and Julian McAuley. 2024. https://arxiv.org/abs/2306.03091 Repobench: Benchmarking repository-level code auto-completion systems . In International Conference on Learning Representations

    2024 arXiv

  12. [20]

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Z...

    2024 arXiv

  13. [21]

    Shuai Lu, Nan Duan, Hojae Han, Daya Guo, Seung-won Hwang, and Alexey Svyatkovskiy. 2022. https://doi.org/10.18653/v1/2022.acl-long.431 R e ACC : A retrieval-augmented code completion framework . In Proceedings of the 60th Annual Meeting of the Association for Computational Lin...

    2022 doi

  14. [22]

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, MING GONG, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shuj...

    2021

  15. [23]

    Tyler McDonnell, Baishakhi Ray, and Miryung Kim. 2013. An empirical study of api stability and adoption in the android ecosystem. In 2013 IEEE International Conference on Software Maintenance, pages 70--79. IEEE

    2013

  16. [24]

    Min, Yangruibo Ding, Luca Buratti, Saurabh Pujar, Gail Kaiser, Suman Jana, and Baishakhi Ray

    Marcus J. Min, Yangruibo Ding, Luca Buratti, Saurabh Pujar, Gail Kaiser, Suman Jana, and Baishakhi Ray. 2024. https://openreview.net/forum?id=caW7LdAALh Beyond accuracy: Evaluating self-consistency of code large language models with identitychain . In The Twelfth International...

    2024

  17. [25]

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. https://openreview.net/forum?id=iaYcJKpY2B_ Codegen: An open large language model for code with multi-turn program synthesis . In The Eleventh International Confe...

    2023

  18. [26]

    OpenAI. 2022. Embedding ada-002. https://platform.openai.com/docs/guides/embeddings/using-embeddings

    2022

  19. [27]

    OpenAI. 2024. Hello gpt-4o. https://openai.com/index/hello-gpt-4o/. [Accessed: October 10, 2024]

    2024

  20. [28]

    Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. 2023. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334

    2023 arXiv

  21. [29]

    Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. 2021. https://openr...

    2021

  22. [30]

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li, Zhiyuan Liu, and Maosong Sun. 2024. https://openreview.net/forum?id=dHng2O0Jjr...

    2024

  23. [31]

    Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval , 3(4):333--389

    2009

  24. [32]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J \'e r \'e my Rapin, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950

    2023 arXiv

  25. [33]

    Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. 2020. Unsupervised translation of programming languages. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA. Curran Associates Inc

    2020

  26. [34]

    Baptiste Roziere, Jie Zhang, Francois Charton, Mark Harman, Gabriel Synnaeve, and Guillaume Lample. 2022. https://openreview.net/forum?id=cmt-6KtR4c4 Leveraging automated unit tests for unsupervised code translation . In International Conference on Learning Representations

    2022

  27. [35]

    Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, Ramesh Nallapati, Murali Krishna Ramanathan, Dan Roth, and Bing Xiang. 2023. https://doi.org/10.18653/v1/2023.acl-long.773 R e C ode: Robust...

    2023 doi

  28. [36]

    Weixiang Yan, Yuchen Tian, Yunzhe Li, Qian Chen, and Wen Wang. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.337 C ode T rans O cean: A comprehensive multilingual benchmark for code translation . In Findings of the Association for Computational Linguistics: EMNLP 2023,...

    2023 doi

  29. [37]

    Daoguang Zan, Bei Chen, Dejian Yang, Zeqi Lin, Minsu Kim, Bei Guan, Yongji Wang, Weizhu Chen, and Jian-Guang Lou. 2022. https://doi.org/10.24963/ijcai.2022/329 Cert: Continual pre-training on sketches for library-oriented code generation . In Proceedings of the Thirty-First In...

    2022 doi

  30. [38]

    Dejiao Zhang, Wasi Ahmad, Ming Tan, Hantian Ding, Ramesh Nallapati, Dan Roth, Xiaofei Ma, and Bing Xiang. 2024. https://openreview.net/forum?id=vfzRRjumpX Codesage: Code representation learning at scale . In The Twelfth International Conference on Learning Representations

    2024

  31. [39]

    Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.151 R epo C oder: Repository-level code completion through iterative retrieval and generation . In Proceedings of t...

    2023 doi

  32. [40]

    Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. 2023. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th AC...

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.