Pith. sign in

REVIEW 4 major objections 6 minor 48 references

Improving FIM Code Completions via Context & Curriculum Based Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Training on hard examples and repo context improves code completion

desk verdict A practical fine-tuning study for FIM code completion with a useful new benchmark and a real A/B test, but the offline results are threatened by a missing train/eval dedup analysis. read the letter →

arxiv 2412.16589 v1 pith:644VNZZV submitted 2024-12-21 cs.IR

Hitesh Sagtani , Rishabh Mehrotra , Beyang Liu This is my paper · ORCID
classification cs.IR
keywords codecompletionfill-in-the-middlecurriculumlearningcontext-awarefine-tuninglanguagemodelsFIMA/Btestingcross-filecontext
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Fill-in-the-middle (FIM) models power code autocomplete, but they are latency-limited to small sizes. This paper claims that fine-tuning such models on a deliberately hard curriculum—code spans at AST node types where completions fail most often—together with the repository context that defines the symbols in those spans, reliably improves completion quality. The gains appear across model families (StarCoder2 and DeepSeek, 1B–7B plus a 2.4B-active MoE), across three offline benchmarks, and in an online A/B test, with the largest relative improvements on the smallest models. If true, this gives latency-constrained products a way to buy accuracy through data curation instead of model size.

What carries the argument

CMFT (Curriculum and Context-aware Fine-Tuning). Curriculum extraction uses tree-sitter to parse code into ASTs, selects node types with low completion-acceptance rates (Call Expression, Function Parameters, If Statement, etc.), ranks them by the number of unique symbols in their subtrees, and samples one node as the completion ground truth. Context extraction then collects the definitions of the symbols in that ground truth—via the TypeScript Compiler API for TypeScript/JavaScript (with nested definitions resolved to depth 2 through a symbol graph) and via BM25 retrieval for other languages—and appends them to the prompt, after removing any snippet that could itself leak the ground truth. Models are LoRA fine-tuned on roughly 200k such examples using the Prefix-Suffix-Middle FIM format, matching each base model's pretraining prompt format.

What would settle it

Compute the overlap between the training repositories and the CCEval and SWE-bench-derived evaluation repositories (e.g., by file-path or content hash), retrain CMFT on a version that explicitly excludes any overlapping repositories, and re-run the same benchmarks; if the Exact Match, Prefix Match, and Pass@1 gains shrink toward the RSFT baseline, the offline claim is explained by data leakage rather than by curriculum and context learning.

Watch

Extended reading notes

Core claim

Fine-tuning FIM code models on a combined dataset of curriculum examples and context examples—denoted CMFT—improves Exact Match on single-line infilling, Prefix Match on CrossCodeEval, and Pass@1 on a new multi-line infilling benchmark, across all tested base models. The paper also reports that in a two-week A/B test on a DeepSeek-Coder-1.6B base, CMFT raised Completion Acceptance Rate by up to 4.24% and Persistence Rate at 30s by up to 2.53% depending on line group, with p75 latency unchanged. The central claim is that the specific composition of fine-tuning data—hard AST-node patterns plus the cross-file context needed to resolve their symbols—is what drives the improvement, not fine-tuning itself, since random-span fine-tuning (RSFT) shows little or no gain.

Load-bearing premise

The load-bearing premise is that the ~200k fine-tuning samples drawn from The Stack and scraped GitHub are effectively disjoint from the repositories used in CCEval and the new SWE-bench-derived benchmark; if those evaluation repos appear in the training data, the reported offline improvements could be memorization rather than genuine completion skill.

Editorial extensions

If this is right

  • CMFT improves Exact Match, Prefix Match, and Pass@1 for StarCoder2 3B/7B, DeepSeek-Coder 1B/7B, and DeepSeek-Coder-v2-lite 2.4B across all three evaluation suites.
  • Smaller models gain the most: Prefix Match on CCEval rises ~6.25% relative for the 1B model versus ~1.25% for the 7B model, so latency-bound deployments can adopt smaller served models.
  • Curriculum and context are complementary: combining them (CMFT) beats either alone (CUFT, COFT), and COFT generally beats CUFT, meaning context utilization is the larger lever.
  • A two-week A/B test on DeepSeek-Coder-1.6B yields statistically significant CAR and CPR gains, with the largest CAR gains (4.24%) on 2–5 line completions and p75 latency maintained.
  • Fine-tuned 7B-scale models reach 13.9–15.5% Pass@1 on the new multi-line infilling benchmark in under 1.2s, versus 19.4–23.4% at 1.67–2.85s for GPT-4o and Claude-3.5-sonnet.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper never reports whether its ~200k fine-tuning samples were deduplicated against the CCEval and SWE-bench-derived evaluation repositories; if public code from those repos leaked into training, the offline gains could be inflated by memorization, and the online A/B results would then be the cleaner evidence.
  • Because context quality is the biggest lever (COFT > CUFT), the method likely transfers best to statically analyzable languages; for languages without a compiler-API equivalent, BM25 retrieval may supply weaker context and smaller gains.
  • The reported offline–online correlation of r=0.6 between Prefix Match and CAR suggests Prefix Match is a usable proxy, but the unexplained 40% of variance leaves room for other factors (e.g., suggestion display position) that offline benchmarks do not capture.
  • A natural testable extension is applying the same curriculum+context recipe to larger chat-based models or to next-line suggestions, where the curriculum node distribution could be re-weighted by online failure rates per language.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a fine-tuning recipe for Fill-in-the-Middle (FIM) code completion models. The authors build a 'curriculum' dataset by extracting AST node types that show low relative Completion Acceptance Rate (CAR) in production telemetry, and a 'context' dataset by retrieving symbol definitions and repository context via the TypeScript compiler API and BM25. They fine-tune StarCoder2 and DeepSeek models with LoRA, comparing Random Span Fine-Tuning (RSFT), Curriculum-Aware Fine-Tuning (CUFT), Context-Aware Fine-Tuning (COFT), and the combined method (CMFT). Offline evaluation is reported on the Single-Line Infilling benchmark, CCEval, and a new Multi-Line Infilling benchmark derived from SWE-bench. A two-week online A/B test on DeepSeek-Coder-1.6B reports significant improvements in CAR and Completion Persistence Rate (CPR) with maintained p75 latency.

Significance. The paper makes a practical, deployment-oriented contribution: it introduces a new public multi-line infilling benchmark, a real-world A/B test with roughly 350k suggestions per arm, and a training-data generation pipeline. If the offline results survive deduplication and statistical testing, the main finding---that context- and curriculum-based fine-tuning helps smaller FIM models without a latency penalty---would be useful both for practitioners and for the code-completion literature. The paper is also commendable for reporting a leakage-control step, though, as detailed below, that step is narrower than the main contamination risk. The central risk is that the offline benchmark table, which carries the cross-architecture and cross-dataset generalization claim, may be inflated by overlap between the ~200k fine-tuning samples and the public evaluation repositories.

major comments (4)
  1. [§4 vs. §5.1] The paper reports no overlap analysis between the fine-tuning corpus (the-stack-dedup plus scraped GitHub, §4) and the evaluation repositories used in CCEval, Single-Line Infilling, and the new SWE-bench-derived Multi-Line Infilling benchmark (§5.1.1, §5.1.2). The leakage control in §4.2 step 6 only removes context snippets that originate from the ground-truth snippet itself; it does not exclude training files whose repository also appears in an evaluation benchmark. Since CCEval and SWE-bench are built from public GitHub repositories and the-stack covers public code, training/eval overlap is a concrete risk. The offline gains in Table 3 (e.g., DeepSeek-Coder-base 1B Python PM 31.5→33.3) could reflect memorization rather than generalization. Please report file-level and repository-level overlap statistics, and ideally re-run the main comparisons on training data deduplicated against all evaluation repositories.
  2. [Table 3 and §5.2.1] The offline results are presented as point estimates without variance, confidence intervals, or significance tests. Many of the reported improvements are small, for example DeepSeek-Coder-base 1B Single-Line Python EM 73.4→73.9 and Starcoder2 7B Single-Line Python EM 66.1→66.4. Without error bars or repeated runs, the caption claim that CMFT 'impoves across all the models architectures, parameters and datasets' is not statistically established. Please add confidence intervals, paired tests, or at least explicit standard errors across multiple fine-tuning seeds.
  3. [Abstract and §5.3.2] The abstract's claim of 'zero latency impact' is under-supported. The online evaluation section only states that the p75 latency metric was maintained, without reporting the actual latency values or a statistical comparison. Table 5 reports offline generation latency for a 256-token generation task, which is not the same as the user-facing p75 latency in the A/B test. Please report the online latency distributions (e.g., p50/p75/p95) for control and treatment arms, and state how 'zero impact' was determined.
  4. [§5.3] The online A/B test uses DeepSeek-Coder-1.6B as the base model, but the offline experiments in Table 3 evaluate DeepSeek-Coder-base 1B, DeepSeek-Coder-v2-lite-base 2.4B, DeepSeek-Coder-base 7B, StarCoder2 3B, and StarCoder2 7B. The 1.6B model does not appear in the offline table. Consequently, the claimed 'directional alignment' between offline and online results is indirect: a different base model was used in the live experiment. Please add offline results for the 1.6B model or temper the generalization claim to reflect that the online validation covers a single, separately evaluated model.
minor comments (6)
  1. [Table 3 caption] The caption contains a typo: 'impoves' should be 'improves'.
  2. [§5.1.4] There is a typo in 'We provider Prompts in the same format...'; it should be 'We provide prompts...'.
  3. [Table 4] In the Starcoder2 7B Python row, COFT reports PM 38.0 and ES 60.1 against CMFT's PM 37.9 and ES 60.0, so the statement that 'CMFT yields the best results across various model families and sizes' is not literally true in every cell; please soften or clarify.
  4. [Figures 2 and 7] 'Relative CAR' is used in Figures 2 and 7 but never formally defined; please specify the normalization (e.g., per-language average) and the denominator used.
  5. [§5.3.2] The Pearson correlation between Prefix Match and CAR is reported as r=0.6, p<0.05, but the number of data points and the unit of analysis (model-language pairs, or something else) are not stated; please provide these details.
  6. [§5.1.3] The StarCoder2 family is described as including a 15B parameter model, but the evaluations only cover the 3B and 7B variants; please state why the 15B model was excluded.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the fine-tuning improvements are empirical results validated on external benchmarks and live A/B tests, with no fitted constants or self-citation chain.

full rationale

The paper's central claims are empirical: fine-tuning FIM models on curriculum and context examples improves completion metrics across external benchmarks (Single-Line Infilling, CCEval, and a new SWE-bench-derived Multi-Line dataset) and in a live A/B test. There is no derivation chain in which an output is defined in terms of an input or a fitted parameter is renamed as a prediction. The curriculum node-type distribution is motivated by Cody CAR telemetry, and CAR is also the online outcome metric; this is a mild selection-overlap concern, but success on those node types after fine-tuning is not logically forced, so it does not constitute circularity. The only self-citation (reference [15]) is contextual related work and is not load-bearing. The data-contamination concern raised about training and evaluation repository overlap is a correctness/validity risk, not a circularity of the paper's argument. Overall, the results are grounded in external benchmarks and live deployment, so no circular step can be exhibited.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper's contribution is empirical; the central results depend on several hand-chosen data generation choices and domain assumptions about what makes code completions hard and what context helps.

free parameters (5)
  • curriculum node-type distribution = percentages in Table 2 (e.g., Call Expression 12%, Random Span 35%)
    Chosen by hand based on internal CAR analysis; affects training data composition and downstream gains.
  • AST node text length quantile threshold = 0.05 to 0.95
    Used to filter extracted nodes; chosen ad hoc in Section 4.1.
  • nested symbol graph depth = 2
    Limits context extraction for nested definitions; set in Section 4.2, Step 4.
  • number of fine-tuning data points = ~200k
    Training dataset size chosen for computational constraints; affects results.
  • LoRA hyperparameters = r=8, alpha=16, batch 16, LR 3e-4, 1 epoch
    Standard but not tuned; may not be optimal per model.
assumptions (4)
  • domain assumption Low CAR AST node types are a valid proxy for example difficulty that curriculum training should target.
    Assumes the correlation between user rejection and intrinsic complexity generalizes to training value; not independently verified.
  • domain assumption Context extracted via TSC/BM25 is beneficial and does not leak ground truth.
    The pipeline removes snippets originating from ground truth, but overall retrieval quality is not measured; Section 4.2.
  • domain assumption Fine-tuning with LoRA preserves and improves FIM capabilities of base models.
    LoRA is used for memory efficiency; parity with full fine-tuning is assumed.
  • domain assumption SWE-bench test reliability filtering ensures ground truth patches are necessary for test passes.
    Section 5.1.2 verifies tests fail without candidate patches, but this assumes test flakiness is negligible.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving FIM Code Completions via Context & Curriculum Based Learning." pith.science (2026). https://pith.science/paper/644VNZZV

@misc{pith2026241216589,
  author       = {Pith},
  title        = {Pith review of: Improving FIM Code Completions via Context & Curriculum Based Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/644VNZZV}},
  note         = {Machine review of arXiv:2412.16589}
}
read the original abstract

Fill-in-the-Middle (FIM) models play a vital role in code completion tasks, leveraging both prefix and suffix context to provide more accurate and contextually relevant suggestions. This paper presents approaches to improve FIM code completion while addressing the challenge of maintaining low latency for real-time coding assistance. We enhance FIM code completion by incorporating context and curriculum examples in the training process. We identify patterns where completion suggestions fail more frequently, revealing complexities that smaller language models struggle with. To address these challenges, we develop a curriculum dataset by extracting hard-to-complete patterns from code repositories and generate context examples using semantic and static analysis tools (e.g. TSC compiler). We fine-tune various sized models, including StarCoder and DeepSeek, on this enhanced dataset. Our evaluation encompasses three key dimensions: the Santa Coder FIM task, the Amazon CCEval benchmark, and a new Multi-Line Infilling evaluation benchmark derived from SWE-bench. Comprehensive ablation studies across multiple model sizes reveal that while all fine-tuned models show improvements, the performance gains are more pronounced for smaller parameter models and incorporating difficult-to-complete examples, as part of curriculum learning, improves the code completion performance. This finding is particularly significant given the latency constraints of code completion tasks. While larger models like GPT and Claude perform well in multi-line completions but are prohibitively challenging to use given high latency, and our fine-tuned models achieve a balance between performance and latency. Finally, we validate our approach through online A/B testing, demonstrating tangible improvements in Completion Acceptance Rate (CAR) and Completion Persistence Rate (CPR), with zero latency impact.

Figures

Figures reproduced from arXiv: 2412.16589 by the authors.

Figure 1
Figure 1. Illustrative examples of various AST node types. The cursor position highlights the position where completions are [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. % Relative CAR at AST nodes: Negative CAR values [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Motivating example for Context learning: Small [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Data generation pipeline illustrating the extraction of curriculum and context examples from source files for [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Prefix Match on CCEval dataset for Python: Im [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Relative CAR (%) at AST nodes: Fine-tuned models [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 13 canonical work pages

  1. [29]

    Vijayaraghavan Murali, Chandra Maddila, Imad Ahmad, Michael Bolin, Daniel Cheng, Negar Ghorbani, Renuka Fernandez, Nachiappan Nagappan, and Peter C Rigby. 2024. AI-assisted Code Authoring at Scale: Fine-tuning, deploying, and mixed methods evaluation. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1066–1085

    2024

  2. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  3. [3]

    Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, Logesh Kumar Umapathi, Carolyn Jane Anderson, Yangtian Zi, Joel Lamy Poirier, Hailey Schoelkopf, Sergey Troshin, Dmitry Abulkhanov, Manuel Romero, Michael Lappert, Francesco De Toni, Bernardo García ...

  4. [4]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021)

    arXiv 2021

  5. [5]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609 (2023)

    arXiv 2023

  6. [6]

    Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. 2022. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207.14255 (2022)

    arXiv 2022

  7. [7]

    Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps- Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. 2023. MultiPL-E: a scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering 49, 7 (2023), 3675–3691

    2023

  8. [8]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

    arXiv 2021

Show all 48 references
  1. [9]

    Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2023. CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion. arXiv:2310.1124...

    2023 arXiv

  2. [10]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  3. [11]

    Omer Dunay, Daniel Cheng, Adam Tait, Parth Thakkar, Peter C Rigby, Andy Chiu, Imad Ahmad, Arun Ganesan, Chandra Maddila, Vijayaraghavan Murali, et al. 2024. Multi-line AI-assisted Code Authoring. In Companion Proceedings of the 32nd ACM International Conference on the Foundati...

    2024

  4. [12]

    Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999 (2022)

    2022 arXiv

  5. [13]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence.arXiv preprint arXiv:2401.14196 (2024)

    2024 arXiv

  6. [14]

    Qi Guo, Xiaohong Li, Xiaofei Xie, Shangqing Liu, Ze Tang, Ruitao Feng, Junjie Wang, Jidong Ge, and Lei Bu. 2024. FT2Ra: A Fine-Tuning-Inspired Approach to Retrieval-Augmented Code Completion. arXiv preprint arXiv:2404.01554 (2024)

    2024 arXiv

  7. [15]

    Jan Hartman, Hitesh Sagtani, Julie Tibshirani, and Rishabh Mehrotra. 2024. AI- assisted Coding with Cody: Lessons from Context Retrieval and Evaluation for Code Recommendations. In Proceedings of the 18th ACM Conference on Recom- mender Systems. 748–750

    2024

  8. [16]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)

    2021 arXiv

  9. [17]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al . 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088 (2024)

    2024 arXiv

  10. [18]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 (2023)

    2023 arXiv

  11. [19]

    Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Car- los Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. 2022. The Stack: 3 TB of permissively licensed source code. arXiv:...

    2022 arXiv

  12. [20]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161 (2023)

    2023 arXiv

  13. [21]

    Tianyang Liu, Canwen Xu, and Julian McAuley. 2023. RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems. arXiv:2306.03091 [cs.CL] https://arxiv.org/abs/2306.03091

    2023 arXiv

  14. [22]

    Wei Liu, Ailun Yu, Daoguang Zan, Bo Shen, Wei Zhang, Haiyan Zhao, Zhi Jin, and Qianxiang Wang. 2024. GraphCoder: Enhancing Repository-Level Code Completion via Code Context Graph-based Retrieval and Language Model. arXiv preprint arXiv:2406.07003 (2024)

    2024 arXiv

  15. [23]

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy- Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. 2024. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173 (2024)

    2024 arXiv

  16. [24]

    Chandra Maddila, Negar Ghorbani, Kosay Jabre, Vijayaraghavan Murali, Edwin Kim, Parth Thakkar, Nikolay Pavlovich Laptev, Olivia Harman, Diana Hsu, Rui Abreu, et al. 2024. AI-Assisted SQL Authoring at Industry Scale. arXiv preprint arXiv:2407.13280 (2024)

    2024 arXiv

  17. [25]

    Xiaofeng Mao, Yufeng Chen, Xiaojun Jia, Rong Zhang, Hui Xue, and Zhao Li

  18. [26]

    Saeed Masoudnia and Reza Ebrahimpour. 2014. Mixture of experts: a literature survey. Artificial Intelligence Review 42 (2014), 275–293

    2014

  19. [27]

    Stoyan Nikolov Maxim Tabachnyk. 2022. ML Enhanced Code Comple- tion. https://research.google/blog/ml-enhanced-code-completion-improves- developer-productivity/

    2022

  20. [28]

    Martin Miksik. 2023. Fine-tuning transformer models for commit message genera- tion and autocompletion. B.S. thesis. University of Twente

    2023

  21. [30]

    Amit Kumar Nandanwar and Jaytrilok Choudhary. 2023. Contextual embeddings- based web page categorization using the fine-tune BERT model. Symmetry 15, 2 (2023), 395

    2023

  22. [31]

    Toan Nguyen-Mau, Anh-Cuong Le, Duc-Hong Pham, and Van-Nam Huynh

  23. [32]

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis.arXiv preprint arXiv:2203.13474 (2022). WSDM ’25, March 10–14, 2025, Hannover, Ge...

    2022 arXiv

  24. [33]

    Huy N Phan, Hoang N Phan, Tien N Nguyen, and Nghi DQ Bui. 2024. Repohyper: Better context retrieval is all you need for repository-level code completion.arXiv preprint arXiv:2403.06095 (2024)

    2024 arXiv

  25. [34]

    Information Fusion 104 (2024), 102202

    An information fusion based approach to context-based fine-tuning of GPT models. Information Fusion 104 (2024), 102202

    2024

  26. [35]

    Matt Post. 2018. A call for clarity in reporting BLEU scores. arXiv preprint arXiv:1804.08771 (2018)

    2018 arXiv

  27. [36]

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundare- san, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297 (2020)

    2020 arXiv

  28. [37]

    Nikhil Pinnaparaju, Reshinth Adithyan, Duy Phung, Jonathan Tow, James Baicoianu, Ashish Datta, Maksym Zhuravinskyi, Dakota Mahan, Marco Bel- lagente, Carlos Riquelme, et al. 2024. Stable code technical report. arXiv preprint arXiv:2404.01226 (2024)

    2024 arXiv

  29. [38]

    Ensheng Shi, Yanlin Wang, Hongyu Zhang, Lun Du, Shi Han, Dongmei Zhang, and Hongbin Sun. 2023. Towards efficient fine-tuning of pre-trained code models: An experimental study and beyond. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Ana...

    2023

  30. [39]

    Disha Shrivastava, Denis Kocetkov, Harm de Vries, Dzmitry Bahdanau, and Torsten Scholak. 2023. Repofusion: Training code models to understand your repository. arXiv preprint arXiv:2306.10998 (2023)

    2023 arXiv

  31. [40]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiao- qing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023)

    2023 arXiv

  32. [41]

    Mistral AI team. 2024. Codestral: Hello, World! https://mistral.ai/news/codestral/

    2024

  33. [42]

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. Codet5: Identifier- aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859 (2021)

    2021 arXiv

  34. [43]

    CodeGemma Team. 2024. Codegemma: Open code models based on gemma. arXiv preprint arXiv:2406.11409 (2024)

    2024 arXiv

  35. [44]

    Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. 2024. Codereval: A benchmark of prag- matic code generation with generative pre-trained models. In Proceedings of the 46th IEEE/ACM International Conference on S...

    2024

  36. [45]

    Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. Repocoder: Repository-level code com- pletion through iterative retrieval and generation.arXiv preprint arXiv:2303.12570 (2023)

    2023 arXiv

  37. [46]

    Di Wu, Wasi Uddin Ahmad, Dejiao Zhang, Murali Krishna Ramanathan, and Xiaofei Ma. 2024. REPOFORMER: Selective retrieval for repository-level code completion. arXiv preprint arXiv:2403.10059 (2024)

    2024 arXiv

  38. [49]

    Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al . 2024. DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence.arXiv preprint arXiv:2406.11931 (2024)

    2024 arXiv

  39. [2023]

    SantaCoder: don’t reach for the stars! arXiv:2301.03988 [cs.SE] https: //arxiv.org/abs/2301.03988

    arXiv

  40. [2024]

    International Journal of Computer Vision 132, 5 (2024), 1685–1700

    Context-aware robust fine-tuning. International Journal of Computer Vision 132, 5 (2024), 1685–1700

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.