Pith. sign in

REVIEW 3 major objections 4 minor 26 references

The way a code-completion system gathers repository context substantially changes output quality, independent of the underlying model.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 11:27 UTC pith:5WZXOAFZ

load-bearing objection The benchmark and competition design are genuinely useful, but the top-ranked results in both tracks are compromised by the paper's own admission that the winning indices may have leaked the reference snippet from future revisions. the 3 major comments →

arxiv 2510.04349 v1 pith:5WZXOAFZ submitted 2025-10-05 cs.SE cs.AIcs.LG

Challenge on Optimization of Context Collection for Code Completion

Dmitry Ustalov , Egor Bogomolov , Alexander Bezzubov , Yaroslav Golubev , Evgeniy Glukhov , Georgii Levtsov , Vladimir Kovalenko This is my paper
classification cs.SE cs.AIcs.LG
keywords code completioncontext collectionfill-in-the-middleretrieval-augmented generationbenchmark datasetchrFPythonKotlin
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper reports a community challenge to answer a narrow but consequential question: given the same code-completion model, how much does the way you gather surrounding code context matter? The organizers built a new benchmark from permissively licensed open-source repositories, using git commit history to separate the context snapshot from the target completion, and asked teams to implement only the context-collection component. In the final private phase, the top strategies combined parsing to find symbol definitions with retrieval over code chunks, and they beat a BM25 baseline by a wide margin on both Python and Kotlin. The paper argues that context collection is therefore a large, separable lever on completion quality, and it releases the full dataset and submissions for reproduction. It also flags in Section IV-G that two top teams indexed all revisions of each repository, which may have leaked future code into the context and inflated the reported gains.

Core claim

On its own terms, the central discovery is that a well-chosen context—symbol definitions and relevant chunks retrieved from the whole repository—yields substantially better fill-in-the-middle completions than the usual recent-file or BM25 baselines, and this holds across Python and Kotlin and across three different coding models. The best private-phase contexts improved average chrF from roughly 0.61 to 0.73 on Python and from 0.63 to 0.75 on Kotlin. Inspection of the six final solutions shows a convergent recipe: parse the code into functions, classes, and symbols; score candidate snippets with a retrieval ranker; and assemble the top hits within the model's token budget. The authors also s

What carries the argument

The central object is the context collector: a black-box module that maps a completion point (prefix, suffix, caret position, and repository snapshot) to a context string. The evaluation protocol fixes the three models and prompt construction, so every difference in the chrF metric is attributable to the context alone. The mechanism that carries the argument is the parse-retrieve-assemble pipeline used by nearly all finalists: extract symbols from the code around the caret, rank candidate definitions or chunks with a retrieval function, and concatenate the top-ranked snippets up to the model's context limit.

Load-bearing premise

The private-phase rankings measure context quality only if the context built from the repository never contains the reference code from later commits; the paper acknowledges in Section IV-G that the top two teams indexed all revisions, which could leak exactly that.

What would settle it

Re-run the private-phase evaluation with the same six containers but with each repository's search index built only from commits strictly before the target commit, then compare the resulting chrF values with the reported ones; if the top teams' scores fall to near the BM25 baseline, the central claim that the winning context collectors are responsible for the gain is false.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Context collection is an independent lever: completion quality can be improved without retraining or enlarging the model, simply by feeding it better repository context.
  • The winning formula—parse symbols, retrieve definitions, assemble within a token budget—provides a strong default design for repository-level completion systems.
  • Because the benchmark separates context collection from inference, future context collectors can be compared directly on public data without rebuilding the model-evaluation pipeline.
  • The fact that similar strategies won on both Python and Kotlin suggests the approach transfers across dynamically and statically typed languages, reducing the need for per-language context engineering.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the temporal-leakage threat is confirmed, the absolute gap between the top teams and baselines may shrink and the final ranking could change; a direct test is to rerun the private-phase containers with index construction restricted to the exact pre-commit snapshot.
  • The same evaluation harness could be reused to study other pipeline components in isolation, such as prompt formatting, context ordering, or token-budget allocation, since the competition shows that isolated component choices have measurable downstream effects.
  • The winning strategies rely on general-purpose retrieval rather than model-specific tuning, which suggests but does not prove that the ranking of context collectors would stay stable for newer or different base models; testing that would be a natural extension.
  • Because chrF compares against one reference completion, a complementary validation would be to check whether top-ranked contexts also improve human-rated usefulness or accept rate on multiple valid completions, not just match a single ground truth.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper reports on a competition organized by JetBrains and Mistral AI at ASE 2025, in which participants designed context-collection strategies for fill-in-the-middle code completion. The task is to extract a context string from a repository snapshot given a completion point (prefix, suffix, caret), with completions generated by three fixed LLMs (Codestral, Qwen2.5-Coder, Mellum) and scored by chrF. The competition had practice, public, and private phases; the private phase used a held-out dataset and ran participant containers offline. The paper describes the dataset (Python and Kotlin repositories, 1,764 completion points), the evaluation protocol, and the submitted solutions. The central empirical claim is that the private-phase rankings (Tables IV and V) identify context-collection strategies that substantially improve completion quality over BM25/recent-file baselines—for example, top Python chrF 0.734 vs. BM25 0.610, and top Kotlin chrF 0.748 vs. BM25 0.634.

Significance. If the private-phase evaluation is valid, the paper provides a useful, reproducible benchmark for repository-level context collection, with a released dataset and a staged protocol that separates context collection from model inference. The multi-model evaluation (three LLMs) and the use of a held-out private phase are genuine strengths, as is the publication of the dataset and participant solution summaries. The paper also honestly documents a threat to validity in Section IV-G. However, the central result—the private-phase ranking—is directly threatened by the admitted leakage risk for the top two teams, and the absence of any leakage-controlled re-analysis or significance testing leaves the headline gains unverified. The paper is therefore a valuable community resource, but its main empirical conclusion needs additional support before it can be accepted as a reliable benchmark result.

major comments (3)
  1. [Section IV-G, Tables IV and V] The paper explicitly admits that the two prize-winning teams, SpareCodeComplete and NoMoreActimel, built retrieval indices over 'all available data in the provided dataset, including other repositories and all the revisions for the given repository.' Because the dataset contains the full git history up to the target commit, indexing later revisions can surface the reference snippet itself in the context. This is not a hypothetical risk: the affected teams occupy ranks 1 and 2 in Python and ranks 1 and 2 in Kotlin. The paper does not quantify how often leakage occurred, does not re-evaluate with leakage-controlled indices (e.g., restricted to the repository snapshot before the target commit), and does not provide a sensitivity analysis. Without this, the reported gains (0.734 vs. 0.610 for Python; 0.748 vs. 0.634 for Kotlin) cannot be attributed to context-collection quality. This is the
  2. [Tables IV and V] No significance testing or confidence intervals are reported. Several ranking differences are small relative to the scale of the metric: e.g., the Kotlin rank-3 tie between WSPR NCSU (0.660) and REALISE Lab (0.659); the Python gap between rank 3 (0.644) and rank 4 (0.636); and the gap between the BM25 baseline (0.634) and Wu Wei (0.627) in Kotlin. Given that chrF is averaged over a few hundred completion points and three models, per-point variance could plausibly change these orderings. The authors should provide per-point bootstrap or permutation-based intervals for the private-phase averages, and should discuss whether the top-tier gaps (e.g., NoMoreActimel vs. SpareCodeComplete in Python) are statistically distinguishable. This is required to support any claim about which strategies are 'best.'
  3. [Section II-C, Section IV-G] The paper states that the LCA-based dataset construction 'allows avoiding the possible temporal data leakages between the context and target completion,' yet Section IV-G shows that the participants' own indexing choices can reintroduce exactly this leakage. The organizers' control over the evaluation is therefore incomplete: in the private phase, participants ran their own code on the provided repository snapshots, and nothing in the protocol prevented them from indexing future revisions. The paper should clarify whether the private-phase containers were sandboxed to prevent access to the full dataset, or whether the organizers verified the retrieved contexts post hoc. If no such verification exists, the claim that the private phase is leak-free (implied by the design description) is not supported.
minor comments (4)
  1. [Table II and III] The column header 'Average chrF↓' uses a downward arrow that may be unintentional; the metric is higher-is-better. Please clarify or remove the arrow to avoid confusion.
  2. [Section II-B] The baseline description is brief. In particular, the 'Recent Files' baseline is described as a proxy for IDE recent files by listing 'other files changed in the same commit,' but it is not clear whether this uses the current commit's changed files (which might include the target file) or only files from the pre-commit snapshot. Please clarify the exact construction.
  3. [Section IV-E and IV-F] The solution summaries would benefit from a short statement of the token budget each team used, since the LLM context window is a shared resource and the paper does not report whether all contexts fit within the smallest model's limit. This is not central, but it would help readers compare the approaches.
  4. [References] Reference [13] is cited for both the competition methodology and the Toloka VQA dataset; consider splitting the citation or adding a separate reference to the original competition description. Also, the Zenodo dataset link [18] is given only as a URL; if a DOI is available, it should be included.

Circularity Check

0 steps flagged

No significant circularity: private-phase rankings are empirical measurements; the admitted future-revision leakage in Section IV-G is a data-integrity threat, not a circular derivation.

full rationale

The paper's central output is an empirical comparison of independent participant context collectors on a held-out private dataset, scored by average chrF of completions from three LLMs (Codestral, Qwen2.5-Coder, and the authors' Mellum). The rankings in Tables IV and V are measurements, not derivations: no parameter is fitted to the private-phase scores and then renamed as a prediction, and no equation defines the target in terms of the inputs. The self-citations to Long Code Arena [6], Toloka VQA [13], and Mellum [12] are methodology or tooling references; the dataset is newly built and split into non-overlapping practice/public/private subsets, so the competition outcome is not forced by those prior works. The paper does, however, self-report a serious validity threat in Section IV-G: 'For building the search indices, the solutions by teams SpareCodeComplete and NoMoreActimel use all available data in the provided dataset, including other repositories and all the revisions for the given repository. Such an approach may lead to a data leakage from the future versions of the same repository, as they may contain the reference code snippet.' This admitted leakage could inflate the top private-phase scores and distort the rankings, but that is an empirical data-contamination issue, not circularity: the leaked reference inflates the metric rather than making the output equal to the input by construction. The correct remedy is a leakage-controlled re-evaluation, not a circularity finding. No circular step satisfies the quoted-reduction standard.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

The paper has no fitted model parameters; its conclusions rest on dataset-construction choices and evaluation assumptions. The load-bearing assumptions are that chrF captures completion quality, that the commit-snapshot design prevents leakage, and that the three-model average is representative. Section IV-G shows the leakage assumption is not guaranteed.

axioms (5)
  • domain assumption chrF is one of the most reliable indicators of code completion quality.
    Adopted from Evtikhiev et al. [14]; the paper does not re-validate this for repository-level fill-in-the-middle completion.
  • domain assumption Using files changed in the same commit is a valid proxy for the IDE 'recent files' context signal.
    Section II-B; the offline competition setting requires a proxy for editor state.
  • domain assumption Taking context from the pre-commit repository snapshot prevents temporal leakage.
    Section II-C, based on Long Code Arena [6]; Section IV-G shows this assumption can be violated when solutions index all revisions.
  • domain assumption Averaging chrF across Codestral, Qwen2.5-Coder, and Mellum is representative of completion quality.
    Section II; no analysis of model-specific variance or agreement is provided.
  • domain assumption The in-house heuristics for selecting multi-line insertions produce realistic completion points.
    Section II-C; the content-based heuristic is not validated against measured developer behavior.

pith-pipeline@v1.3.0-alltime-deepseek · 9566 in / 11146 out tokens · 83678 ms · 2026-08-04T11:27:57.665603+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Challenge on Optimization of Context Collection for Code Completion." pith.science (2026). https://pith.science/paper/5WZXOAFZ

@misc{pith2026251004349,
  author       = {Pith},
  title        = {Pith review of: Challenge on Optimization of Context Collection for Code Completion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5WZXOAFZ}},
  note         = {Machine review of arXiv:2510.04349}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The rapid advancement of workflows and methods for software engineering using AI emphasizes the need for a systematic evaluation and analysis of their ability to leverage information from entire projects, particularly in large code bases. In this challenge on optimization of context collection for code completion, organized by JetBrains in collaboration with Mistral AI as part of the ASE 2025 conference, participants developed efficient mechanisms for collecting context from source code repositories to improve fill-in-the-middle code completions for Python and Kotlin. We constructed a large dataset of real-world code in these two programming languages using permissively licensed open-source projects. The submissions were evaluated based on their ability to maximize completion quality for multiple state-of-the-art neural models using the chrF metric. During the public phase of the competition, nineteen teams submitted solutions to the Python track and eight teams submitted solutions to the Kotlin track. In the private phase, six teams competed, of which five submitted papers to the workshop.

Figures

Figures reproduced from arXiv: 2510.04349 by Alexander Bezzubov, Dmitry Ustalov, Egor Bogomolov, Evgeniy Glukhov, Georgii Levtsov, Vladimir Kovalenko, Yaroslav Golubev.

Figure 1
Figure 1. Figure 1: When a completion is requested, the IDE gathers the context at [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The simplest context collection: no context clues, only some of the [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 6 linked inside Pith

  1. [1]

    CodeFill: Multi-Token Code Completion by Jointly Learning from Structure and Naming Sequences,

    M. Izadi, R. Gismondi, and G. Gousios, “CodeFill: Multi-Token Code Completion by Jointly Learning from Structure and Naming Sequences,” inProceedings of the 44th International Conference on Software Engi- neering, 2022, pp. 401–412

    2022

  2. [2]

    Efficient Training of Language Models to Fill in the Middle,

    M. Bavarian, H. Jun, N. Tezak, J. Schulman, C. McLeavey, J. Tworek, and M. Chen, “Efficient Training of Language Models to Fill in the Middle,”arXiv preprint arXiv:2207.14255, 2022

    Pith/arXiv arXiv 2022

  3. [3]

    RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation,

    F. Zhang, B. Chen, Y . Zhang, J. Keung, J. Liu, D. Zan, Y . Mao, J.- G. Lou, and W. Chen, “RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation,” inProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 2471–2484

    2023

  4. [4]

    Code Llama: Open Foundation Models for Code,

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remezet al., “Code Llama: Open Foundation Models for Code,”arXiv preprint arXiv:2308.12950, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    Repo- former: Selective Retrieval for Repository-Level Code Completion,

    D. Wu, W. U. Ahmad, D. Zhang, M. K. Ramanathan, and X. Ma, “Repo- former: Selective Retrieval for Repository-Level Code Completion,” in Proceedings of the 41st International Conference on Machine Learning, vol. 235, 2024, pp. 53 270–53 290

    2024

  6. [6]

    Long Code Arena: A Set of Benchmarks for Long-Context Code Models,

    E. Bogomolov, A. Eliseeva, T. Galimzyanov, E. Glukhov, A. Shapkin, M. Tigina, Y . Golubev, A. Kovrigin, A. Van Deursen, M. Izadiet al., “Long Code Arena: A Set of Benchmarks for Long-Context Code Models,”arXiv preprint arXiv:2406.11612, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    On Pretraining For Project-Level Code Completion,

    M. Sapronov and E. Glukhov, “On Pretraining For Project-Level Code Completion,” inICLR 2025 Third Workshop on Deep Learning for Code, 2025

    2025

  8. [8]

    RLCoder: Reinforcement Learning for Repository-Level Code Com- pletion,

    Y . Wang, Y . Wang, D. Guo, J. Chen, R. Zhang, Y . Ma, and Z. Zheng, “RLCoder: Reinforcement Learning for Repository-Level Code Com- pletion,” in2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), 2025, pp. 1140–1152

    2025

  9. [9]

    Repohyper: Search-Expand-Refine on Semantic Graphs for Repository-Level Code Completion,

    H. N. Phan, H. N. Phan, T. N. Nguyen, and N. D. Bui, “Repohyper: Search-Expand-Refine on Semantic Graphs for Repository-Level Code Completion,” in2025 IEEE/ACM Second International Conference on AI Foundation Models and Software Engineering (Forge), 2025, pp. 14– 25

    2025

  10. [10]

    Codestral,

    Mistral AI, “Codestral,” 2025, Mistral AI. [Online]. Available: https://mistral.ai/news/codestral

    2025

  11. [11]

    Qwen2.5-Coder Technical Report,

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Luet al., “Qwen2.5-Coder Technical Report,”arXiv preprint arXiv:2409.12186, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    Introducing Mellum: JetBrains’ New LLM Built for Developers,

    JetBrains, “Introducing Mellum: JetBrains’ New LLM Built for Developers,” 2024, JetBrains Blog. [Online]. Avail- able: https://blog.jetbrains.com/blog/2024/10/22/introducing-mellum- jetbrains-new-llm-built-for-developers/

    2024

  13. [13]

    Toloka visual question answering benchmark,

    D. Ustalov, N. Pavlichenko, S. Koshelev, D. Likhobaba, and A. Smirnova, “Toloka visual question answering benchmark,”arXiv preprint arXiv:2309.16511, 2023

    Pith/arXiv arXiv 2023

  14. [14]

    Out of the BLEU: How Should We Assess Quality of the Code Generation Models?

    M. Evtikhiev, E. Bogomolov, Y . Sokolov, and T. Bryksin, “Out of the BLEU: How Should We Assess Quality of the Code Generation Models?”Journal of Systems and Software, vol. 203, p. 111741, 2023

    2023

  15. [15]

    chrF: Character n-gram F-score for Automatic MT Eval- uation,

    M. Popovi ´c, “chrF: Character n-gram F-score for Automatic MT Eval- uation,” inProceedings of the Tenth Workshop on Statistical Machine Translation, 2015, pp. 392–395

    2015

  16. [16]

    From Commit Message Generation to History-Aware Commit Message Completion,

    A. Eliseeva, Y . Sokolov, E. Bogomolov, Y . Golubev, D. Dig, and T. Bryksin, “From Commit Message Generation to History-Aware Commit Message Completion,” in2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2023, pp. 723– 735

    2023

  17. [17]

    The Probabilistic Relevance Framework: BM25 and Beyond,

    S. Robertson and H. Zaragoza, “The Probabilistic Relevance Framework: BM25 and Beyond,”Foundations and Trends in Information Retrieval, vol. 3, no. 4, pp. 333–389, 2009

    2009

  18. [18]

    Dataset for the ASE 2025 Challenge on Code Completion Context Collection Optimization

    JetBrains, “Dataset for the ASE 2025 Challenge on Code Completion Context Collection Optimization.” [Online]. Available: https://zenodo. org/records/16964765

    arXiv 2025

  19. [19]

    Evalai: Towards better evaluation systems for ai agents,

    D. Yadav, R. Jain, H. Agrawal, P. Chattopadhyay, T. Singh, A. Jain, S. B. Singh, S. Lee, and D. Batra, “Evalai: Towards better evaluation systems for ai agents,”arXiv preprint arXiv:1902.03570, 2019

    Pith/arXiv arXiv 1902

  20. [20]

    Exploration of Structural Code Relationship Space for Context Collection,

    C. Sofianos, “Exploration of Structural Code Relationship Space for Context Collection,” inProceedings of the 40th IEEE/ACM International Conference on Automated Software Engineering Workshops, 2025

    2025

  21. [21]

    Billion-scale Similarity Search with GPUs,

    J. Johnson, M. Douze, and H. J ´egou, “Billion-scale Similarity Search with GPUs,”IEEE Transactions on Big Data, vol. 7, no. 3, pp. 535–547, 2019

    2019

  22. [22]

    Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,

    N. Reimers and I. Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,” inProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing (EMNLP- IJCNLP), 2019, pp. 3982–3992

    2019

  23. [23]

    Relative Positioning Based Code Chunking Method For Rich Context Retrieval In Repository Level Code Completion Task With Code Language Model,

    I. Rahman and M. R. Rahman, “Relative Positioning Based Code Chunking Method For Rich Context Retrieval In Repository Level Code Completion Task With Code Language Model,” inProceedings of the 40th IEEE/ACM International Conference on Automated Software Engineering Workshops, 2025

    2025

  24. [24]

    Beyond More Context: How Granularity and Order Drive Code Completion Quality,

    U. Yusuf, G. Caumartin, and D. E. Costa, “Beyond More Context: How Granularity and Order Drive Code Completion Quality,” inProceedings of the 40th IEEE/ACM International Conference on Automated Software Engineering Workshops, 2025

    2025

  25. [25]

    On the Importance of Con- text Filtering in Retrieval-Augmented Code Completion,

    S. Sedov, V . Savinskiy, and A. Arzhantsev, “On the Importance of Con- text Filtering in Retrieval-Augmented Code Completion,” inProceedings of the 40th IEEE/ACM International Conference on Automated Software Engineering Workshops, 2025

    2025

  26. [26]

    SpareCodeSearch: Searching for Code Context When You Have No Spare GPU,

    M. Nguyen, “SpareCodeSearch: Searching for Code Context When You Have No Spare GPU,” inProceedings of the 40th IEEE/ACM Inter- national Conference on Automated Software Engineering Workshops, 2025

    2025