Pith. sign in

REVIEW 4 major objections 6 minor 28 references

Structure-Aware Corpus Construction and User-Perception-Aligned Metrics for Large-Language-Model Code Completion

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The longest common prefix between a suggested completion and the final code, not exact or partial overlap, predicts whether developers accept the suggestion.

desk verdict The metric validation is circular—the reference answer is constructed from user acceptance—so the headline claim about LCP is not established; the corpus method is plausible but under-benchmarked. read the letter →

arxiv 2505.13073 v1 pith:KFUWSXFS submitted 2025-05-19 cs.SE cs.AI

Dengfeng Liu , Jucai Zhai , Xiaoguang Jiang , Ziqun Li , Qianjin Yu , Feng Liu , Rui Ye , Huang Liu
show 3 more authors
Zhiguo Yang Yongsheng Du Fang Tan
This is my paper · ORCID
classification cs.SEcs.AI
keywords codecompletionevaluationmetricsuserperceptionlongestcommonprefixROUGE-LCPknowledgegraphcross-filedependencyabstractsyntaxtree
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that user acceptance of an on-the-fly code completion is governed by the length of the exact prefix run that matches the final code, not by overall overlap. To capture this, it proposes LCP and its length-normalized variant ROUGE-LCP, and reports that across more than ten thousand logged completion events from a commercial code assistant, LCP correlates with the daily adoption rate more strongly than exact match, ROUGE-L, or LCS. The same paper argues that repository-level completion accuracy improves when the training corpus is rebuilt around AST-complete semantic units and reordered along directed call and reference paths, a construction it calls the SPSR-Graph. If these claims hold, evaluation of code assistants can move toward prefix-focused metrics that mirror how developers actually edit.

What carries the argument

LCP is the number of consecutive characters, starting at the first character, that a suggested completion shares with the reference; ROUGE-LCP divides that count by the reference length $|R|$ so samples of different sizes can be compared. The probability model treats the first mismatch as the end of a success run, giving $P(n=k)$ as the product of conditional match probabilities up to $k$ times one minus the conditional mismatch probability at the first error position, which yields the long-tail shape the logs exhibit. The second mechanism is the SPSR-Graph: a directed graph whose nodes are AST-extracted semantic units, such as functions, structs, and classes, and whose edges encode calls, references, type usages, macro expansions, and file includes. Training samples are built by concatenating nodes along breadth-first paths up to a maximum depth $D$, with file-path comments inserted at cross-file boundaries, so the model sees dependency order explicitly during pretraining.

What would settle it

Recompute the daily correlation using a reference that is not derived from the accepted suggestion, such as expert-authored expected completions or the user's final code from sessions where suggestions were not accepted, and check whether LCP still predicts adoption; if the correlation falls to near zero, the claimed perception alignment is an artifact of the logging definition.

Watch

Extended reading notes

Core claim

The central discovery is that the longest common prefix (LCP) between a suggested completion and the reference text is the best logged predictor of whether a developer presses Tab to adopt the suggestion. The paper models LCP as a long-tail distribution: the run of correct prefix characters survives only while each conditional next-character probability holds, and the first mismatch ends it. Empirically, LCP's Pearson correlation with daily adoption rate was above 0.7 across the full two-month window, beating LCS, ROUGE-L, exact match, and its own normalized variant ROUGE-LCP. The paper also claims that its SPSR-Graph corpus construction, which reorders code into samples that follow function-level and struct-level dependency paths, raises exact-match and BLEU scores on the on-the-fly completion task while keeping prefix quality stable.

Load-bearing premise

The logged reference answer is defined as whatever the user ended up with after pressing Enter, so accepted suggestions write their own prefix into the reference; the measured link between longer common prefixes and adoption therefore partly follows from how the data was collected, not from an independent property of the metric.

Editorial extensions

If this is right

  • Fine-tuning a completion model with a loss aligned to LCP should increase the fraction of suggestions whose opening run is correct, which is the behavior users actually reward with Tab.
  • ROUGE-LCP normalizes by reference length, making it usable for fair comparisons across completions of different lengths, unlike raw LCP.
  • Replacing token-level Fill-in-the-Middle masking with AST-complete subtree masking preserves structure without hurting prefix quality, since LCP stays stable while exact match rises.
  • Adding function-level and struct-level code-graph reordering to the pretraining corpus improves C and C++ completion, with the largest exact-match gain from function-level graphs in C.
  • Graph traversal breadth should be kept near the observed dependency count, about 4, because wider traversal can add irrelevant context and reduce accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the logging artifact is removed, with reference answers taken from independent ground truth rather than from what the user finally typed, and LCP still tracks adoption, the metric becomes a cheap always-on training signal for live code assistants; that test is not in the paper.
  • The SPSR-Graph construction is a pretraining-time analogue of retrieval-augmented generation; a natural extension would be to retrieve the same graph paths at inference time and compare latency and accuracy against pretraining-only exposure.
  • The observed LCP long tail suggests most accepted completions are short correct prefixes, so product decisions about when to trigger a suggestion may matter more than maximizing whole-line exact match; this follows from the paper's data but is not stated by the authors.
  • A controlled experiment that inserts syntactically valid but semantically wrong tokens at different prefix positions could separate LCP's sensitivity to the first error from ROUGE-L's sensitivity to total overlap, which the current logged data cannot disentangle.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper makes two contributions for on-the-fly code completion with large language models. First, it proposes LCP and ROUGE-LCP as evaluation metrics that are claimed to align with user perception, supported by a probabilistic derivation in Sec. 3.2 and by Pearson correlations with user adoption rates computed from ZTE-Code-Copilot logs in Sec. 4.2. Second, it proposes SPSR-Graph, a structure-preserving, semantically reordered code graph for building repository-level training corpora via AST-based semantic unit extraction and graph traversal, evaluated in Sec. 4.4 by fine-tuning Qwen2.5-7B-Coder on C/C++ corpora and comparing pipeline variants.

Significance. If the metric-alignment claim were valid, the paper would provide a practical, easy-to-compute alternative to EM and ROUGE-L for evaluating code completion in interactive settings, and the SPSR-Graph pipeline would be a plausible engineering contribution for repository-level pretraining. The paper deserves credit for collecting a real user-behavior dataset and for attempting to connect metric design to a probabilistic model of prefix correctness. However, the central validation is compromised by the data collection protocol: the reference answer is defined as the user-confirmed final content, which is directly influenced by whether the user accepted the suggestion. As a consequence, the observed correlations between LCP and adoption rate are partly mechanical and do not establish that LCP reflects user perception independently of the logging definition. The theoretical derivation is essentially the definition of first-error probability and does not connect to adoption behavior. The corpus experiments also lack external baselines and significance testing. The paper's core claim is therefore not supported as written.

major comments (4)
  1. [§4.1, §4.2, Tables 1–2] The reference answer used for computing LCP is defined as 'the content confirmed by the user pressing the Enter key' (§4.1). In an on-the-fly completion session, if the user presses Tab, the suggestion text is inserted into the editor buffer, so the final Enter-confirmed content inherits the suggestion's prefix. Consequently, LCP(S,R) is inflated exactly for the cases labeled as adopted and suppressed for rejected suggestions. The correlations in Tables 1 and 2 (r > 0.6–0.9) therefore follow in part from the logging definition rather than from an intrinsic property of LCP as a perception-aligned metric. The comparison with EM, LCS, and ROUGE-L in Table 2 is computed against the same contaminated reference, so the claim that LCP is superior for capturing user perception is not established independently of the data collection protocol.
  2. [§3.2, Eq. (1)] The derived expression for P(n=k) is the standard definition of the probability that the first error occurs at position k+1: it is the product of conditional probabilities of correct prefixes followed by the first mismatch. The derivation does not introduce any model of user behavior, so it does not 'explain the relationship between these metrics and user adoption behavior' as claimed in §3.2. Moreover, the assumption that the reference R is a fixed target sequence is violated in the logged data, where R is generated after S has been presented, as noted in the previous comment.
  3. [§3.2, piecewise formula for ROUGE-LCP] The piecewise definition of P(Rouge-LCP(S,R)) is internally inconsistent with the definition ROUGE-LCP(S,R)=LCP(S,R)/|R|. In the third case, where LCP(R,S)=|R| and S≠R, the formula writes (LCP(S,R)+|S_ext|)/|R|, which can exceed 1, contradicting the preceding definition that ROUGE-LCP never exceeds 1. Additionally, the notation switches between LCP(S,R) and LCP(R,S) without clarification. Please reconcile the definition and the distributional formula, and clarify which argument order is intended for a symmetric evaluation metric.
  4. [§4.4, Table 3, §4.5] The experiments for the SPSR-Graph contribution compare only the authors' own pipeline variants (Pipeline, +AST, +KGF, +KGF S). There is no comparison with existing repository-level completion methods such as CoCoMIC, RepoFusion, or GraphCoder, which are cited in §2.2, and no statistical significance tests or variance estimates are reported. The improvements in Table 3 are small (e.g., LCP stays at 5.2 for C across all conditions), and Fig. 5 is presented without error bars or multiple runs, so the claim of 'significant improvement' is not supported.
minor comments (6)
  1. [Table 3] The header 'Blue' should be 'BLEU' for consistency with the rest of the paper.
  2. [§5] In the conclusion, 'we proposes a data processing method' should be 'we propose a data processing method'.
  3. [Algorithm 1] The pseudocode formatting is confusing: the line 'S←∅ ;' appears to be part of the first line, and the function definitions are not clearly separated from the main body. Please restructure for readability.
  4. [Figure 4 and Figure 8 captions] The abbreviations 'R-L' and 'R-LCP' are used in the heatmaps but defined only indirectly in the caption; define them explicitly in each caption.
  5. [§3.2] The statement 'According to the Central Limit Theorem, the length of the reference text |R| typically follows an approximately Gaussian distribution in real-world settings' is asserted without justification and is not used later. If it is meant to support the mixed-distribution claim, explain how the CLT applies to code lengths; otherwise remove it.
  6. [§4.5] The phrase 'the traversal depth is fixed at 1' combined with the stated complexity O(n + nd + n·d^D·m) in §3.3.2 is unclear; clarify whether the complexity formula is intended for depth D and how the breadth parameter k is incorporated into the traversal.

Circularity Check

1 steps flagged · score 6.0 of 10

LCP–adoption correlation is partly built into the logging definition because the reference answer is the Enter-confirmed buffer content; the SPSR-Graph results are not circular.

  1. self definitional [Sec. 3.1 (Definition of Adoption), Sec. 4.1 (Experimental Setup, reference definition), Tables 1-2]
    "The logged information includes ... reference answers (i.e., the content confirmed by the user pressing the Enter key). / Definition of Adoption: Developers accept the code suggestions provided by Copilot and directly apply them to the project, either as-is or after modification. Here, we consider each press of the 'Tab' key by the user as one adoption."

    LCP(S,R) is computed between the model's suggestion S and the logged reference R. R is defined as the buffer content at the moment the user presses Enter. If the user earlier pressed Tab, the suggestion (or a prefix of it) has been inserted into that buffer, so for adopted samples R contains a prefix of S by construction, inflating LCP. For rejected samples R is user-typed code with no required relation to S, so LCP is low. The reported Pearson correlations in Tables 1 and 2 therefore partly encode the logging protocol's coupling between Tab-adoption and Enter-confirmation, rather than an independent user-perception property of LCP. The Sec. 3.2 probability model assumes R is independent 'actual code', which the logged R is not.

full rationale

The paper's central metric-validation claim is partially circular. The authors define adoption as a Tab press and define the logged reference answer as the content confirmed by pressing Enter. In an interactive completion session, pressing Tab inserts the suggestion into the editor, so the final reference for adopted suggestions inherits a prefix of the suggestion by construction; LCP is therefore mechanically inflated exactly for the adopted cases and depressed for rejected cases. This coupling makes the reported correlation between LCP and adoption rate, and the claimed superiority of LCP/ROUGE-LCP over EM/ROUGE-L/LCS, partly an artifact of the data collection definition rather than an independently established property of the metric. The theoretical derivation in Sec. 3.2 does not remove this issue because it treats R as independent ground-truth code rather than as post-acceptance buffer content. However, this is not a full 10: user behavior could still produce or break such correlations through later edits, and the paper's SPSR-Graph corpus-construction experiments (Sec. 4.4-4.5) are independent empirical comparisons with no evident circularity. No load-bearing self-citation chain or ansatz-smuggling was found. The score reflects that one central empirical claim reduces in part to the logging definition, while the corpus method and its evaluations remain self-contained.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central claims rest on several domain assumptions about user behavior and ground truth, plus hyperparameters that are not fully specified. The most consequential assumption is the independence of the logged reference answer, which is contradicted by the data collection process.

free parameters (4)
  • Granularity control parameter theta (AST FIM) = not specified
    Controls the maximum size of AST subtrees selected for masking; chosen by hand in Sec 3.3.1 and Appx B.
  • Graph traversal depth D = 1
    Fixed at 1 in Sec 4.5 because the task targets single-line or inline completion.
  • Graph breadth k = 3-7
    Maximum expansion breadth during graph path enumeration; selected based on average dependency count and model input length (Sec 4.5).
  • Data filtering thresholds (line lengths, character ratios, etc.) = not specified
    Stated as rules in Appx A.1 with no numeric values, making the preprocessing pipeline underspecified.
assumptions (4)
  • domain assumption Adoption rate (Tab presses) is a valid proxy for user satisfaction with a code suggestion.
    The paper equates user perception with adoption rate throughout (Sec 3.1).
  • domain assumption The logged reference answer (content confirmed by Enter) is independent ground truth for completion quality.
    Used to compute LCP and ROUGE-LCP; in fact the reference is influenced by the user's acceptance decision (Sec 4.1).
  • ad hoc to paper The length of the reference text |R| is approximately Gaussian by the Central Limit Theorem.
    Invoked in Sec 3.2 to motivate a mixed distribution, but no argument that code lengths are i.i.d.; the statement is not used in any quantitative prediction.
  • domain assumption Graph-based reordering along call paths improves a model's cross-module understanding.
    The implicit assumption behind SPSR-Graph; tested only against the authors' own ablations (Sec 4.4).
invented entities (1)
  • SPSR-Graph (Structure-Preserving and Semantically-Reordered Code Graph)
    purpose: Model function-level call and struct reference relationships across files to construct training samples that preserve cross-module dependencies.
    Introduced in this paper; validated only through internal ablations, with no external benchmark or reproducible artifact.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Structure-Aware Corpus Construction and User-Perception-Aligned Metrics for Large-Language-Model Code Completion." pith.science (2026). https://pith.science/paper/KFUWSXFS

@misc{pith2026250513073,
  author       = {Pith},
  title        = {Pith review of: Structure-Aware Corpus Construction and User-Perception-Aligned Metrics for Large-Language-Model Code Completion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KFUWSXFS}},
  note         = {Machine review of arXiv:2505.13073}
}
read the original abstract

Code completion technology based on large language model has significantly improved the development efficiency of programmers. However, in practical applications, there remains a gap between current commonly used code completion evaluation metrics and users' actual perception. To address this issue, we propose two evaluation metrics for code completion tasks--LCP and ROUGE-LCP, from the perspective of probabilistic modeling. Furthermore, to tackle the lack of effective structural semantic modeling and cross-module dependency information in LLMs for repository-level code completion scenarios, we propose a data processing method based on a Structure-Preserving and Semantically-Reordered Code Graph (SPSR-Graph). Through theoretical analysis and experimental validation, we demonstrate the superiority of the proposed evaluation metrics in terms of user perception consistency, as well as the effectiveness of the data processing method in enhancing model performance.

Figures

Figures reproduced from arXiv: 2505.13073 by the authors.

Figure 1
Figure 1. Overview of the proposed framework. The framework includes three stages: corpus preprocessing, AST [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. LCP Distribution and Its Relationship with [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. ROUGE-LCP Distribution and Its Relation [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Daily Metric and Adoption Rate Distributions, Heatmap of Correlation Between Evaluation Metrics and [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Impact of code knowledge graph breadth on [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: LCP Distribution and Its Relationship with Adoption Count and Adoption Rate [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: ROUGE-LCP Distribution and Its Relationship with Adoption Count and Adoption Rate [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Daily Metric and Adoption Rate Distributions, Heatmap of Correlation Between Evaluation Metrics and [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 5 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai - Wei Chang. 2021. https://doi.org/10.18653/V1/2021.NAACL-MAIN.211 Unified pre-training for program understanding and generation . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, On...

    doi:10.18653/v1/2021.naacl-main.211 2021

  4. [4]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. https://arxiv.org/abs/2108.07732 Program synthesis with large language models . Preprint, arXiv:2108.07732

    arXiv 2021

  5. [5]

    Gareth Ari Aye, Seohyun Kim, and Hongyu Li. 2021. https://doi.org/10.1109/ICSE-SEIP52600.2021.00022 Learning autocompletion from real-world datasets . In Proceedings of the 43rd International Conference on Software Engineering: Software Engineering in Practice, ICSE-SEIP '21, page 131–139. IEEE Press

    arXiv 2021

  6. [6]

    Vitaliy Bibaev, Alexey Kalina, Vadim Lomshakov, Yaroslav Golubev, Alexander Bezzubov, Nikita Povarov, and Timofey Bryksin. 2022. https://doi.org/10.1145/3540250.3558968 All you need is logs: improving code completion by learning from anonymous ide usage logs . In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on t...

    arXiv 2022

  7. [7]

    Cursor. 2023. https://www.cursor.com https://www.cursor.com

    work page 2023

  8. [8]

    Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2023. https://arxiv.org/pdf/2310.11248.pdf Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion . In Thirty-seventh Conference on Neural Information Proces...

    arXiv 2023

Show all 28 references
  1. [9]

    Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2024. https://aclanthology.org/2024.lrec-main.305 Cocomic: Code completion by jointly modeling in-file and cross-file context . In Proceeding...

    2024

  2. [10]

    GitHub. 2021. https://github.com/features/copilot https://github.com/features/copilot

    2021

  3. [11]

    Linyuan Gong, Mostafa Elhoushi, and Alvin Cheung. 2024. https://openreview.net/forum?id=cBWVJh5Fvf AST-T5: structure-aware pretraining for code generation and understanding . In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2...

    2024

  4. [12]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, and 5 others. 2024. https://arxiv.org/abs/2409.12186 Qwe...

    2024 arXiv

  5. [13]

    Maliheh Izadi, Jonathan Katzy, Tim Van Dam, Marc Otten, Razvan Mihai Popescu, and Arie Van Deursen. 2024. https://doi.org/10.1145/3597503.3639138 Language models for code completion: A practical evaluation . In Proceedings of the IEEE/ACM 46th International Conference on Softw...

    2024

  6. [14]

    Siyuan Jiang, Jia Li, He Zong, Huanyu Liu, Hao Zhu, Shukai Hu, Erlu Li, Jiazheng Ding, Yu Han, Wei Ning, Gen Wang, Yihong Dong, Kechi Zhang, and Ge Li. 2025. https://arxiv.org/abs/2410.13187 aixcoder-7b: A lightweight and effective large language model for code processing . Pr...

    2025 arXiv

  7. [15]

    Jia Li, Hao Zhu, Huanyu Liu, Xianjie Shi, He Zong, Yihong Dong, Kechi Zhang, Siyuan Jiang, Zhi Jin, and Ge Li. 2025. https://arxiv.org/abs/2503.15301 aixcoder-7b-v2: Training llms to fully utilize the long context in repository-level code completion . Preprint, arXiv:2503.15301

    2025

  8. [16]

    Junwei Liu, Yixuan Chen, Mingwei Liu, Xin Peng, and Yiling Lou. 2024 a . Stall+: Boosting llm-based repository-level code completion with static analysis. arXiv preprint arXiv:2406.10018

    2024 arXiv

  9. [17]

    Wei Liu, Ailun Yu, Daoguang Zan, Bo Shen, Wei Zhang, Haiyan Zhao, Zhi Jin, and Qianxiang Wang. 2024 b . https://doi.org/10.48550/ARXIV.2406.07003 Graphcoder: Enhancing repository-level code completion via code context graph-based retrieval and language model . CoRR, abs/2406.07003

    2024 doi arXiv

  10. [18]

    Mengnan Qi, Yufan Huang, Yongqiang Yao, Maoquan Wang, Bin Gu, and Neel Sundaresan. 2024. https://arxiv.org/abs/2404.08885 Is next token prediction sufficient for gpt? exploration on code logic comprehension . Preprint, arXiv:2404.08885

    2024 arXiv

  11. [19]

    Chi, Nathanael Sch \" a rli, and Denny Zhou

    Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Sch \" a rli, and Denny Zhou. 2023. https://proceedings.mlr.press/v202/shi23a.html Large language models can be easily distracted by irrelevant context . In International Conference on Mac...

    2023

  12. [20]

    Disha Shrivastava, Denis Kocetkov, Harm de Vries, Dzmitry Bahdanau, and Torsten Scholak. 2023. https://doi.org/10.48550/ARXIV.2306.10998 Repofusion: Training code models to understand your repository . CoRR, abs/2306.10998

    2023 doi arXiv

  13. [21]

    Wannita Takerngsaksiri, Chakkrit Tantithamthavorn, and Yuan-Fang Li. 2024. https://doi.org/10.1016/j.infsof.2023.107336 Syntax-aware on-the-fly code completion . Inf. Softw. Technol., 165(C)

    2024

  14. [22]

    Tim van Dam, Maliheh Izadi, and Arie van Deursen. 2023. https://doi.org/10.1109/MSR59073.2023.00035 Enriching source code with contextual data for code completion models: An empirical study . In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR),...

    2023

  15. [23]

    Tim Van Dam, Frank Van der Heijden, Philippe De Bekker, Berend Nieuwschepen, Marc Otten, and Maliheh Izadi. 2024. Investigating the performance of language models for completing code in functional programming languages: a haskell case study. In Proceedings of the 2024 IEEE/ACM...

    2024

  16. [24]

    Yanlin Wang, Yanli Wang, Daya Guo, Jiachi Chen, Ruikai Zhang, Yuchi Ma, and Zibin Zheng. 2024. https://arxiv.org/abs/2407.19487 Rlcoder: Reinforcement learning for repository-level code completion . Preprint, arXiv:2407.19487

    2024 arXiv

  17. [25]

    Joty, and Steven C

    Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. 2021. https://doi.org/10.18653/V1/2021.EMNLP-MAIN.685 Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation . In Proceedings of the 2021 Conference on Empirical Met...

    2021 doi

  18. [26]

    Qinyun Wu, Chao Peng, Pengfei Gao, Ruida Hu, Haoyu Gan, Bo Jiang, Jinhe Tang, Zhiwen Deng, Zhanming Guan, Cuiyun Gao, and 1 others. 2024. Repomastereval: Evaluating code completion via real-world repositories. arXiv preprint arXiv:2408.03519

    2024

  19. [27]

    Jian Yang, Jiaxi Yang, Ke Jin, Yibo Miao, Lei Zhang, Liqun Yang, Zeyu Cui, Yichang Zhang, Binyuan Hui, and Junyang Lin. 2024. https://arxiv.org/abs/2412.05210 Evaluating and aligning codellms on human preference . Preprint, arXiv:2412.05210

    2024 arXiv

  20. [28]

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, and 1 others. 2024. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint ar...

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.