Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Structure-Aware Fill-in-the-Middle Pretraining for Code

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AST-FIM: syntax-aware fill-in-the-middle pretraining outperforms random span masking for code models.

desk verdict A well-controlled pretraining study where syntax-aware FIM beats random-character FIM, but the missing token-FIM baseline and the 90/10 mix make the 'syntax matters' claim not yet proven. read the letter →

arxiv 2506.00204 v1 pith:LH5LN2VC submitted 2025-05-30 cs.CL cs.AIcs.SE

Linyuan Gong , Alvin Cheung , Mostafa Elhoushi , Sida Wang This is my paper
classification cs.CLcs.AIcs.SE
keywords abstractsyntaxtreesfill-in-the-middlepretrainingcodelanguagemodelssyntax-awaremaskinginfillingbenchmarkGitHubcommitsTree-sitterediting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that fill-in-the-middle (FIM) pretraining for code language models should mask complete syntactic structures rather than random character spans. The authors propose AST-FIM, which parses training code with Tree-sitter and selects masked spans that align with Abstract Syntax Tree subtrees, using a mix of single-node and aligned-span masking. They report that this syntax-aware objective makes models better at real-world code completion and editing: across 1B and 8B parameter models, AST-FIM outperforms random-character FIM on the SAFIM benchmark and on their new Real-FIM-Eval benchmark built from more than 30,000 GitHub commits in 12 languages. They also report that AST-FIM preserves left-to-right generation performance at a 0.7 FIM rate, where random FIM degrades it. If correct, the paper shows that respecting code structure during pretraining is a better default than treating code as plain text.

What carries the argument

The load-bearing object is the masking algorithm: instead of picking an arbitrary character span, AST-FIM selects a span that exactly covers one or more complete subtrees of the code's parse tree. Single-Node Masking samples one AST node with probability proportional to its byte size; Aligned-Span Masking draws a random character span, finds the lowest subtree containing it, and then chooses the contiguous run of that subtree's children with the largest character-level Intersection over Union with the sampled span. Training then follows the standard FIM recipe, rearranging prefix, suffix, and middle with sentinel tokens using PSM and SPM formats in equal proportion, and combines with a left-to-right objective at a 70% FIM rate, with 90% of FIM examples using AST-FIM and 10% using random FIM to preserve random-span capability.

What would settle it

Compute the fraction of changed code in a large sample of real git commits whose boundaries exactly match AST subtree boundaries. If most edits are partial statements, renames inside expressions, or reformatted fragments that cut across subtrees, then a benchmark built from those edits should show AST-FIM at or below random-FIM performance.

Watch

Extended reading notes

Core claim

The paper's central claim is that making the masked middle of FIM training examples align with complete Abstract Syntax Tree subtrees improves a code model's infilling ability on real-world edits while keeping standard generation intact. On the SAFIM benchmark, AST-FIM beats the Rand-FIM baseline on every subtask, and on Real-FIM-Eval it achieves lower perplexity on both the Add and Edit splits. The same pattern holds at 1B and 8B scale, and the 8B model trained on 2T tokens is competitive with publicly available base models of similar size on HumanEval+ and MBPP+ while leading them on infilling. The paper also introduces Real-FIM-Eval, a benchmark whose examples are drawn from real git commits, treating added or updated code as the infill target.

Load-bearing premise

The method's advantage rests on real-world code edits generally deleting or replacing complete syntactic units rather than arbitrary character spans; if edits routinely split AST subtrees, the syntax prior will misalign with actual editing patterns.

Editorial extensions

If this is right

  • Syntax-aware masking becomes a stronger default than random-character masking for code FIM pretraining, since it improves infilling on both structure-aware and real-edit benchmarks.
  • Code models can be trained at high FIM rates such as 0.7 without sacrificing left-to-right HumanEval+ and MBPP+ performance, reducing the trade-off that previously forced lower FIM rates.
  • The method is language-agnostic, so any language with a Tree-sitter grammar can receive the same syntax-aware objective without per-language engineering.
  • Models trained with AST-FIM reach a given infilling quality after fewer pretraining tokens than Rand-FIM, roughly 50 to 70 percent of the tokens.
  • Base models pretrained this way are stronger starting points for code completion and editing systems than models pretrained with random FIM.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the advantage is driven by structural coherence rather than span length, grammar-aware masking might transfer beyond code to other structured artifacts such as JSON, configuration files, or mathematical expressions.
  • A clean test of the mechanism would compare AST-FIM against token-level and line-level FIM with matched middle-length distributions; the paper does not report this comparison, so it remains open whether complete-subtree boundaries or simply cleaner span boundaries drive the gain.
  • Real-FIM-Eval treats the full added code segment as the infill target, which makes the benchmark sympathetic to complete-subtree masks; an edit benchmark that samples partial-line edits or renames inside expressions would probe the boundary of that assumption.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes AST-FIM, a fill-in-the-middle (FIM) pretraining objective for decoder-only code models that masks spans aligned with abstract-syntax-tree (AST) subtrees rather than random character spans. It also introduces Real-FIM-Eval, a benchmark derived from 30,000+ GitHub commits across 12 languages, with Add and Edit splits evaluated by character-level perplexity. The authors train 1B and 8B Llama-3 models from scratch under controlled settings and report that AST-FIM outperforms a random-character FIM baseline on SAFIM and Real-FIM-Eval while preserving left-to-right (L2R) performance on HumanEval+ and MBPP+, unlike the random-FIM baseline. They also compare an 8B AST-FIM model against existing base models of similar size.

Significance. If the findings hold, the paper makes a useful contribution: it shows that syntax-aware span selection can be incorporated into decoder-only FIM pretraining at scale in a language-agnostic way, and it provides a realistic commit-derived FIM benchmark. The strengths are the controlled experimental setup (identical architecture, data, and compute for the main comparisons), training from scratch at 1B and 8B scales, public code and benchmark release, and a limitations section that honestly acknowledges several gaps. The Real-FIM-Eval benchmark addresses a real gap in FIM evaluation, though the choice of character-level perplexity as the only metric limits its conclusiveness.

major comments (4)
  1. [Section 5.1, Table 1] The main comparison in Table 1 is against a random-character FIM baseline only, while the dominant FIM variants in current code LLMs (e.g., StarCoder, DeepSeek-Coder) use token-level or line-level span masking. Section 7 acknowledges that 'A more careful test of random FIM, token FIM, and line FIM with tuned lengths is needed.' This is load-bearing for the central claim that syntax-aware masking is a better default than standard random FIM; without token-FIM or line-FIM baselines under identical training settings, the reported gains over character-level FIM could be due simply to masking larger or more coherent spans rather than to syntax awareness. I request at least a 1B-scale comparison against token-FIM and line-FIM baselines with matched FIM rates and span-length distributions.
  2. [Section 5.1] The paper states that within the FIM portion, the AST-FIM model uses 90% AST-FIM and 10% Rand-FIM, while the Rand-FIM baseline uses 100% random spans. This design confounds the masking strategy with the data mix, and no ablation with pure AST-FIM (0% random mix) against pure Rand-FIM is reported. Without that ablation, it is unclear whether the observed benefit comes from syntax-aware masking specifically or from training on a mixture of two masking distributions, which could provide useful diversity. This is a load-bearing gap for attributing the gains to AST structure.
  3. [Section 6.1, Tables 1-2] The reported Pass@1 and perplexity differences are not accompanied by error bars or significance tests. For example, on HumanEval+ (164 examples) the AST-FIM vs Rand-FIM gap is 15.9 vs 11.6 (4.3 points), which is within typical binomial confidence intervals for a sample of this size; the MBPP+ gap (35.3 vs 31.5 on 371 examples) is similarly small. The statement that Rand-FIM 'experiences a significant performance decrease' is therefore unsupported. Likewise, Real-FIM-Eval perplexity differences such as 1.230 vs 1.232 on the Edit split for the 1B model are tiny. Confidence intervals, multiple seeds, or another form of uncertainty quantification are needed to support the claims of consistency and significance.
  4. [Section 4.2] Real-FIM-Eval is evaluated using character-level perplexity only, which measures the likelihood of the ground-truth middle given the prefix and suffix rather than the quality of the generated infill. The paper itself lists generative metrics as future work. While perplexity is a reasonable low-noise diagnostic, the central claim of 'better performance on real-world FIM tasks' requires a generative evaluation (e.g., greedy-decoding exact match or edit distance) to be fully convincing. Additionally, since a core premise is that real-world edits involve complete syntactic units (Section 3.3), the paper should analyze what fraction of Real-FIM-Eval middles actually correspond to complete AST subtrees; otherwise the benchmark may be biased in favor of AST-FIM by construction, weakening the real-world evidence.
minor comments (4)
  1. [Section 3.3] The sentence 'Find the lowest 3 AST subtree' contains a stray '3' that appears to be a footnote marker placed in the wrong position; it should be removed or repositioned.
  2. [Section 5.1] The term 'context-level FIM' is used without a definition; please clarify how it differs from document-level FIM and describe how the context window interacts with span selection.
  3. [Section A.1] In the 'SPM without special tokens' prompt format, the text 'suffixêêprefix middle' uses 'êê' in place of what should likely be two line breaks; this should be corrected in the final version.
  4. [Abstract and Section 2] The abstract and related work claim support for '100+ programming languages,' but the experiments cover only 12 languages; please either temper the claim or report parse-success statistics across a broader set of languages to substantiate it.

Circularity Check

0 steps flagged · score 2.0 of 10

No circularity: controlled pretraining runs and a commit-derived benchmark support the central claim; the same-author SAFIM benchmark is aligned with the training objective but is not load-bearing.

full rationale

The paper's derivation chain is empirical, not formal. AST-FIM is a training-data masking algorithm; Rand-FIM and L2R baselines are trained under identical architectures, data, and compute budgets, so the comparisons in Tables 1-2 and Figure 2 are controlled experiments rather than quantities fitted to the claimed outcomes. Real-FIM-Eval is constructed from GitHub commit diffs and does not restrict the masked middle to AST subtrees, so the benchmark does not encode AST-FIM's definition; the advantage on Real-FIM-Eval is therefore an independent measurement. The only self-referential element is the use of SAFIM (Gong et al., 2024b), a benchmark authored by the same group and explicitly designed to test syntax-aware AST-subtree completion. The paper itself acknowledges the resulting distribution match: "Better matching training and testing distributions still result in stronger performance for various FIM tasks and offer a good explanation why AST-FIM is strong on SAFIM and Real-FIM" (Section 7). Because SAFIM's construction aligns with AST-FIM's masking distribution, the SAFIM result is an expected consistency check rather than load-bearing evidence for real-world editing, and the independent Real-FIM-Eval result supplies the external evidence. The absence of token-FIM and line-FIM baselines is acknowledged ("A more careful test of random FIM, token FIM, and line FIM with tuned lengths is needed"), but missing baselines are a completeness/correctness concern, not circularity. No fitted parameter is relabeled as a prediction, no equation reduces to its own input, and no uniqueness claim is imported from a same-author citation. I therefore find no significant circularity; a minor self-benchmark/self-citation issue keeps the score at 2 rather than 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. The special tokens [PRE], [SUF], [MID] are from prior FIM work. The main assumptions are about parsing validity, the alignment of real edits with AST subtrees, and the validity of the evaluation metric.

free parameters (4)
  • FIM rate = 0.7
    Fraction of training steps using the FIM objective, chosen by the authors; affects the trade-off between FIM and L2R performance.
  • AST-FIM proportion within FIM = 0.9
    90% of FIM examples use AST-FIM and 10% use Rand-FIM; a manual design choice to keep exposure to random spans.
  • PSM/SPM mix = 0.5/0.5
    Equal mix of prefix-suffix-middle and suffix-prefix-middle FIM formats, standard in prior FIM work.
  • Single-Node / Aligned-Span mix = 0.5/0.5
    Final model uses a 50:50 mix; the ablation in Table 5 shows this combination performs best, so the configuration is selected based on evaluation results.
assumptions (5)
  • domain assumption Code files in the pretraining corpus are syntactically valid, allowing Tree-sitter parsing.
    Section 3.3 states this assumption and reports a 98.2% parse success rate on GitHub code; files that fail to parse fall back to Rand-FIM.
  • domain assumption Real-world code edits align with complete AST subtrees.
    Core premise of AST-FIM introduced in Section 3.3; if false, the masking bias would not match developer editing patterns.
  • domain assumption Tree-sitter ASTs capture meaningful code structure for infilling tasks.
    The method relies on Tree-sitter's concrete syntax trees as a proxy for abstract syntax; the paper notes the distinction is not important for this purpose.
  • domain assumption Character-level perplexity on the middle span is a valid evaluation metric for FIM quality.
    Section 4.2 adopts perplexity for low noise, but it does not measure functional correctness or edit similarity; the paper acknowledges generative metrics as future work.
  • domain assumption The Real-FIM-Eval benchmark is free from data contamination with pretraining data.
    Section 4.1 states the benchmark period (Jan-Feb 2025) is separate from pretraining data, but overlap cannot be fully ruled out.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Structure-Aware Fill-in-the-Middle Pretraining for Code." pith.science (2026). https://pith.science/paper/LH5LN2VC

@misc{pith2026250600204,
  author       = {Pith},
  title        = {Pith review of: Structure-Aware Fill-in-the-Middle Pretraining for Code},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LH5LN2VC}},
  note         = {Machine review of arXiv:2506.00204}
}
read the original abstract

Fill-in-the-Middle (FIM) is a common pretraining method for code LLMs, where models complete code segments given surrounding context. However, existing LLMs treat code as plain text and mask random character spans. We propose and evaluate AST-FIM, a pretraining strategy that leverages Abstract Syntax Trees (ASTs) to mask complete syntactic structures at scale, ensuring coherent training examples better aligned with universal code structures and common code editing patterns such as blocks, expressions, or functions. To evaluate real-world fill-in-the-middle (FIM) programming tasks, we introduce Real-FIM-Eval, a benchmark derived from 30,000+ GitHub commits across 12 languages. On infilling tasks, experiments on 1B and 8B parameter models show that AST-FIM is particularly beneficial for real-world code editing as it outperforms standard random-character FIM by up to 5 pts on standard FIM benchmarks. Our code is publicly available at https://github.com/gonglinyuan/ast_fim.

Figures

Figures reproduced from arXiv: 2506.00204 by the authors.

Figure 1
Figure 1. Comparison of masking strategies in Random-Character FIM (Rand-FIM) and our proposed AST-Aware FIM (AST-FIM) in two examples. The highlighted code is the masked part for FIM training. Left: Rand-FIM treats code as a character sequence, masking a random span. Right: AST-FIM respects code structure by masking complete subtrees. This syntax-aware masking aligns more closely with typical developer-code interactions. 0 2… view at source ↗
Figure 2
Figure 2. Performance of 1B parameter model during different types of pretraining, check￾pointed every 4000 steps (16.7B tokens). Left: Pass@1 of MBPP+, a left-to-right task (higher is better). Middle: Average pass@1 of SAFIM-Algorithm, SAFIM-Control, and SAFIM-API (higher is better). Right: Average perplexity of Real-FIM-Eval-Add and Real-FIM-Eval-Edit (lower is better). further language specific engineering. This approach m… view at source ↗
Figure 3
Figure 3. Comparison of training inputs processed by Rand-FIM and AST-FIM using the PSM format. Given the same code, Rand-FIM selects a random character span as the “middle” part, while AST-FIM selects a span corresponding to entire AST subtrees. AST-FIM generates cleaner training examples that better reflect practical code completion scenarios. 4 [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Construction of Fill-in-the-Middle (FIM) examples for the proposed Real-FIM-Eval benchmark splits, derived from real-world git commits. Add: Uses code insertions; the added code becomes the “middle” to predict. Edit: Uses code modifications, presented via a conflict-me…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models

    cs.AI 2026-07 unverdicted novelty 6.0 of 10

    Mid-training on function-aware fill-in-the-middle data improves SWE-Bench by 2.8-5.4 points and reduces post-training capability loss.

Reference graph

Works this paper leans on

31 extracted references · 4 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    S anta C oder: don’t reach for the stars! 0 (arXiv:2301.03988), February 2023

    Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, Logesh Kumar Umapathi, Carolyn Jane Anderson, Yangtian Zi, Joel Lamy Poirier, Hailey Schoelkopf, Sergey Troshin, Dmitry Abulkhanov, Manuel Romero, Michael Lappert, Francesco De Toni, Bernardo García ...

    arXiv doi:10.48550/arxiv.2301.03988 2023

  3. [3]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. Aug 2021. doi:10.48550/arXiv.2108.07732. http://arxiv.org/abs/2108.07732. arXiv:2108.07732 [cs]

    arXiv doi:10.48550/arxiv.2108.07732 2021

  4. [4]

    Efficient training of language models to fill in the middle

    Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. 0 (arXiv:2207.14255), July 2022. doi:10.48550/arXiv.2207.14255. http://arxiv.org/abs/2207.14255. arXiv:2207.14255 [cs]

    arXiv doi:10.48550/arxiv.2207.14255 2022

  5. [5]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    arXiv doi:10.48550/arxiv.2005.14165 2020

  6. [6]

    Tree‑sitter: An incremental parsing system for programming tools, 2025

    Max Brunsfeld and Tree‑Sitter Contributors. Tree‑sitter: An incremental parsing system for programming tools, 2025. https://github.com/tree-sitter/tree-sitter

    work page 2025

  7. [7]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    arXiv doi:10.48550/arxiv.2107.03374 2021

  8. [8]

    Copilot arena: A platform for code LLM evaluation in the wild

    Wayne Chi, Valerie Chen, Anastasios Nikolas Angelopoulos, Wei-Lin Chiang, Aditya Mittal, Naman Jain, Tianjun Zhang, Ion Stoica, Chris Donahue, and Ameet Talwalkar. Copilot arena: A platform for code LLM evaluation in the wild. 0 (arXiv:2502.09328), February 2025. doi:10.48550/arXiv.2502.09328. http://arxiv.org/abs/2502.09328. arXiv:2502.09328

    arXiv doi:10.48550/arxiv.2502.09328 2025

Show all 31 references
  1. [9]

    C ode G emma: Open code models based on G emma

    CodeGemma Team . C ode G emma: Open code models based on G emma. 0 (arXiv:2406.11409), June 2024. doi:10.48550/arXiv.2406.11409. http://arxiv.org/abs/2406.11409. arXiv:2406.11409

    2024 doi arXiv

  2. [10]

    BERT : Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT : Pre-training of deep bidirectional transformers for language understanding. May 2019. doi:10.48550/arXiv.1810.04805. http://arxiv.org/abs/1810.04805. arXiv:1810.04805 [cs]

    2019 doi arXiv

  3. [11]

    Horizon-length prediction: Advancing fill-in-the-middle capabilities for code generation with lookahead planning

    Yifeng Ding, Hantian Ding, Shiqi Wang, Qing Sun, Varun Kumar, and Zijian Wang. Horizon-length prediction: Advancing fill-in-the-middle capabilities for code generation with lookahead planning. 2024. https://arxiv.org/abs/2410.03103

    2024

  4. [12]

    Enabling language models to fill in the blanks

    Chris Donahue, Mina Lee, and Percy Liang. Enabling language models to fill in the blanks. 0 (arXiv:2005.05339), September 2020. doi:10.48550/arXiv.2005.05339. http://arxiv.org/abs/2005.05339. arXiv:2005.05339

    2005 doi arXiv

  5. [13]

    I n C oder: A generative model for code infilling and synthesis

    Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. I n C oder: A generative model for code infilling and synthesis. Apr 2023. doi:10.48550/arXiv.2204.05999. http://arxiv.org/abs/2204.05999....

    2023 doi arXiv

  6. [14]

    AST - T 5: Structure-aware pretraining for code generation and understanding

    Linyuan Gong, Mostafa Elhoushi, and Alvin Cheung. AST - T 5: Structure-aware pretraining for code generation and understanding. 0 (arXiv:2401.03003), January 2024 a . doi:10.48550/arXiv.2401.03003. http://arxiv.org/abs/2401.03003. arXiv:2401.03003 [cs]

    2024 doi arXiv

  7. [15]

    Evaluation of LLM s on syntax-aware code fill-in-the-middle tasks

    Linyuan Gong, Sida Wang, Mostafa Elhoushi, and Alvin Cheung. Evaluation of LLM s on syntax-aware code fill-in-the-middle tasks. 0 (arXiv:2403.04814), June 2024 b . doi:10.48550/arXiv.2403.04814. http://arxiv.org/abs/2403.04814. arXiv:2403.04814

    2024 doi arXiv

  8. [16]

    Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I. Wang. CRUXE val: A benchmark for code reasoning, understanding and execution. 0 (arXiv:2401.03065), January 2024. doi:10.48550/arXiv.2401.03065. http://arxiv.org/abs/2401.03065. arXiv:...

    2024 doi arXiv

  9. [17]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. D eep S eek- C oder: When the large language model meets programming -- the rise of code intelligence. 0 (arXiv:2401.14196...

    2024 doi arXiv

  10. [18]

    Qwen2.5- C oder technical report

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou...

    2024 doi arXiv

  11. [19]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE - B ench: Can language models resolve real-world G ithub issues? 0 (arXiv:2310.06770), October 2023. doi:10.48550/arXiv.2310.06770. http://arxiv.org/abs/2310.06770. a...

    2023 doi arXiv

  12. [20]

    Weld, Luke Zettlemoyer, and Omer Levy

    Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. S pan BERT : Improving pre-training by representing and predicting spans. 0 (arXiv:1907.10529), January 2020. doi:10.48550/arXiv.1907.10529. http://arxiv.org/abs/1907.10529. arXiv:1907.10529

    1907 doi arXiv

  13. [21]

    Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan ...

    2025 doi arXiv

  14. [22]

    S tar C oder: may the source be with you! May 2023

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, ...

    2023 doi arXiv

  15. [23]

    Is your code generated by chat GPT really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chat GPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023 a . https://openrevi...

    2023

  16. [24]

    R epo B ench: Benchmarking repository-level code auto-completion systems

    Tianyang Liu, Canwen Xu, and Julian McAuley. R epo B ench: Benchmarking repository-level code auto-completion systems. 0 (arXiv:2306.03091), October 2023 b . doi:10.48550/arXiv.2306.03091. http://arxiv.org/abs/2306.03091. arXiv:2306.03091 [cs]

    2023 doi arXiv

  17. [25]

    S tar C oder 2 and the stack v2: The next generation

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Z...

    2024 arXiv

  18. [26]

    Singh, Rylan Schaeffer, Andrew Poulton, Sanmi Koyejo, Pontus Stenetorp, Sharan Narang, and Dieuwke Hupkes

    Lovish Madaan, Aaditya K. Singh, Rylan Schaeffer, Andrew Poulton, Sanmi Koyejo, Pontus Stenetorp, Sharan Narang, and Dieuwke Hupkes. Quantifying variance in evaluation benchmarks. 0 (arXiv:2406.10229), June 2024. doi:10.48550/arXiv.2406.10229. http://arxiv.org/abs/2406.10229. ...

    2024 doi arXiv

  19. [27]

    https://mistral.ai/news/codestral-2501

    MistralAI, 2025. https://mistral.ai/news/codestral-2501

    2025

  20. [28]

    The impact of AI on developer productivity: Evidence from G ithub C opilot

    Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. The impact of AI on developer productivity: Evidence from G ithub C opilot. 0 (arXiv:2302.06590), February 2023. doi:10.48550/arXiv.2302.06590. http://arxiv.org/abs/2302.06590. arXiv:2302.06590

    2023 doi arXiv

  21. [29]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Jul 2020. doi:10.48550/arXiv.1910.10683. http://arxiv.org/abs/1910.1...

    2020 doi arXiv

  22. [30]

    Code llama: Open foundation models for code

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défoss...

    2023 doi arXiv

  23. [31]

    Wang, Alex Gu, Lovish Madaan, Dieuwke Hupkes, Jiawei Liu, Yuxiang Wei, Naman Jain, Yuhang Lai, Sten Sootla, Ofir Press, Baptiste Rozière, and Gabriel Synnaeve

    Sida I. Wang, Alex Gu, Lovish Madaan, Dieuwke Hupkes, Jiawei Liu, Yuxiang Wei, Naman Jain, Yuhang Lai, Sten Sootla, Ofir Press, Baptiste Rozière, and Gabriel Synnaeve. E val- A rena: noise and errors on llm evaluations. https://github.com/crux-eval/eval-arena, 2024

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.