pith. sign in

arxiv: 2310.11248 · v2 · pith:POIDNBDGnew · submitted 2023-10-17 · 💻 cs.LG · cs.CL· cs.SE

CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion

Yangruibo Ding , Zijian Wang , Wasi Uddin Ahmad , Hantian Ding , Ming Tan , Nihal Jain , Murali Krishna Ramanathan , Ramesh Nallapati
show 3 more authors
Parminder Bhatia Dan Roth Bing Xiang
This is my paper
classification 💻 cs.LG cs.CLcs.SE
keywords codecross-filecontextcompletioncrosscodeevaldiversebenchmarkcapability
0
0 comments X
read the original abstract

Code completion models have made significant progress in recent years, yet current popular evaluation datasets, such as HumanEval and MBPP, predominantly focus on code completion tasks within a single file. This over-simplified setting falls short of representing the real-world software development scenario where repositories span multiple files with numerous cross-file dependencies, and accessing and understanding cross-file context is often required to complete the code correctly. To fill in this gap, we propose CrossCodeEval, a diverse and multilingual code completion benchmark that necessitates an in-depth cross-file contextual understanding to complete the code accurately. CrossCodeEval is built on a diverse set of real-world, open-sourced, permissively-licensed repositories in four popular programming languages: Python, Java, TypeScript, and C#. To create examples that strictly require cross-file context for accurate completion, we propose a straightforward yet efficient static-analysis-based approach to pinpoint the use of cross-file context within the current file. Extensive experiments on state-of-the-art code language models like CodeGen and StarCoder demonstrate that CrossCodeEval is extremely challenging when the relevant cross-file context is absent, and we see clear improvements when adding these context into the prompt. However, despite such improvements, the pinnacle of performance remains notably unattained even with the highest-performing model, indicating that CrossCodeEval is also capable of assessing model's capability in leveraging extensive context to make better code completion. Finally, we benchmarked various methods in retrieving cross-file context, and show that CrossCodeEval can also be used to measure the capability of code retrievers.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. BIRDS: Characterizing and Understanding Biodiversity Impact of Large Language Model Serving

    q-bio.OT 2026-05 unverdicted novelty 7.0

    BIRDS framework quantifies request-level biodiversity impacts of LLM serving via operational and embodied pathways and introduces QNBI to jointly assess impact and quality, showing accumulation at scale across workloa...

  2. When Retrieval Hurts Code Completion: A Diagnostic Study of Stale Repository Context

    cs.SE 2026-05 accept novelty 7.0

    Stale repository context in code RAG actively induces models to produce obsolete helper references, raising stale outputs by 76-88 percentage points over current-only retrieval in a 17-sample diagnostic study.

  3. Benchmarking Requirement-to-Architecture Generation with Hybrid Evaluation

    cs.SE 2026-04 unverdicted novelty 7.0

    R2ABench benchmark shows LLMs generate syntactically valid software architectures from requirements but produce structurally fragmented results due to weak relational reasoning.

  4. Code2LoRA: Hypernetwork-Generated Adapters for Code Language Models under Software Evolution

    cs.SE 2026-06 unverdicted novelty 6.0

    Code2LoRA generates repo-specific LoRA adapters via hypernetwork for code LMs, matching per-repo LoRA on static tasks and exceeding shared LoRA by 5.2 pp on evolving code in a 604-repo benchmark.

  5. ATM: CID-Brokered Pre-Write Admission for Multi-Agent Code Co-Synthesis

    cs.SE 2026-06 unverdicted novelty 5.0

    ATM is a CID-brokered governance framework that maps write intents to semantic atoms for pre-admission control, validation, and neutral-steward application in single-domain multi-agent code synthesis.

  6. How Does Chunking Affect Retrieval-Augmented Code Completion? A Controlled Empirical Study

    cs.SE 2026-05 conditional novelty 5.0

    Function-based chunking underperforms other strategies in RAG code completion by 3.57-5.64 points, with context length as the dominant factor.

  7. LLM-Oriented Information Retrieval: A Denoising-First Perspective

    cs.IR 2026-05 unverdicted novelty 5.0

    Denoising to maximize usable evidence density and verifiability is becoming the primary bottleneck in LLM-oriented information retrieval, conceptualized via a four-stage framework and addressed through a pipeline taxo...

  8. LLM-Oriented Information Retrieval: A Denoising-First Perspective

    cs.IR 2026-05 unverdicted novelty 4.0

    Argues for a denoising-first paradigm in LLM-oriented information retrieval, framing challenges via a four-stage progression and providing a taxonomy of signal-to-noise optimization techniques across the pipeline.