Pith. sign in

REVIEW 3 major objections 4 minor 47 references

Mid-training on whole-function fill-in-the-middle examples improves coding agents' issue-resolution accuracy and curbs the skill erosion that agentic post-training alone causes.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 06:28 UTC pith:LB4N4GZH

load-bearing objection A genuinely new FIM mid-training recipe with consistent gains and honest ablations, but the missing control against generic continued pretraining leaves the central causal claim unproven, and the 7B headline gain rests on a weakened baseline. the 3 major comments →

arxiv 2607.12463 v3 pith:LB4N4GZH submitted 2026-07-14 cs.AI cs.CL

Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models

Yubo Wang , Jiarong Liang , Yuxuan Zhang , Xuye Liu , Cong Wei , Yuyu Zhang , Ping Nie , Wenhu Chen This is my paper
classification cs.AI cs.CL
keywords fill-in-the-middlemid-trainingcoding agentsprogram dependency graphchain-of-thoughtself-supervised learningcapability preservationsoftware engineering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that the step-by-step loop of a coding agent—history, action, external tool return, continuation—is structurally the same as a function call site, where a caller binds arguments, a callee computes a value, and downstream code consumes it. The authors exploit this with a self-supervised 'function-aware fill-in-the-middle' training stage, applied to ordinary Python code before the usual agentic post-training. Across several model sizes and two different post-training pipelines, this stage improves performance on issue-resolution benchmarks by a few points, and it reduces the hidden capability erosion that agentic post-training alone inflicts on general coding and tool-use skills. The paper argues that a cheap, self-supervised stage can install an agent-relevant inductive bias before any expensive trajectory data is introduced.

Core claim

The paper's central claim is that mid-training a code model to reconstruct whole functions from their surrounding caller and callee context—using a fill-in-the-middle objective, with a chain-of-thought rationale placed inside the masked region—makes that model a better coding agent after subsequent agent-specific post-training. The evidence is a set of consistent gains on a standard software-engineering issue-resolution benchmark: a few percentage points across two model sizes, across two different agentic post-training pipelines, and on a different base model. The gains concentrate on tasks whose gold patches touch multiple functions in the same file, and the training also reverses most of

What carries the argument

The central mechanism is the claimed structural isomorphism between a function call site and a single agent step: both decompose into context, call/action, return/observation, and continuation. The paper operationalizes this with function-aware fill-in-the-middle (FIM): masked targets are whole functions or small connected groups of functions, selected by building a program dependency graph from a file's abstract syntax tree and scoring each candidate with two hand-designed quantities—a complexity score (lines of code, cyclomatic complexity, nesting depth) and an inferability score (how much of the body can be recovered from callers, callees, signature, docstring, and class siblings). A chai

Load-bearing premise

The load-bearing premise is that the measured improvements come from the function-call inductive bias specifically, rather than from merely training on 2.6 billion additional tokens of Python code; the paper controls for rationale source, selection method, and masking granularity, but does not compare against an equal-budget training run with random masking or plain next-token prediction on the same corpus.

What would settle it

Train the same 2.6-billion-token corpus with a random-span fill-in-the-middle objective (or ordinary left-to-right next-token prediction) at the same compute budget and with the same post-training pipeline; if the issue-resolution gains and the recovery of non-coding skills match the function-aware variant, then the dependency-graph selection and function-level masking are not the operative cause. Alternatively, mid-train on code stripped of function calls and check whether the transfer to non-coding tool-use benchmarks disappears.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Coding-agent capabilities can be improved with self-supervised data that already exists in ordinary code repositories, without collecting new human or synthetic agent trajectories.
  • Agentic post-training should be preceded by a stage that teaches the model to condition on externally produced returns; the benefit survives the later post-training.
  • The gain concentrating on multi-function tasks implies that a large part of issue-resolution difficulty is cross-function dependency tracking, and that function-level masking targets exactly this difficulty.
  • The near-elimination of no-patch failures suggests that FIM-style training instills a disposition to produce a non-empty continuation, counteracting a common premature-stop failure mode in agents.
  • The transfer to non-coding tool-use benchmarks implies that the function-call inductive bias generalizes to any 'call something external and consume the result' behavior, not just code editing.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The isomorphism argument suggests a testable extension: mid-training on other structured call-return formats (API docs, config files, even natural-language descriptions of functions) might yield similar agent gains, since the mechanism is about conditioning on externally produced values rather than about Python syntax.
  • Because the selection algorithm is hand-designed and the paper shows selection is the dominant lever, a learned selection criterion—e.g., one based on the model's own prediction uncertainty—could plausibly push the gains further; this is not tested in the paper.
  • The capability-regression numbers imply that current agentic post-training pipelines pay a hidden general-skills cost that is underreported; if the same cost exists in other domains, a cheap mid-training stage that restores it could become a standard component of agent training.
  • Since the corpus is Python-only yet transfers to tool use, a natural prediction is that multilingual or multi-paradigm function-fill data would transfer even more broadly, for example to terminal control or web navigation tasks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a self-supervised mid-training stage for coding-agent foundation models: function-aware fill-in-the-middle (FIM) on a 2.6B-token decontaminated Python corpus, with mask targets selected by program dependency graph analysis and a complexity–inferability double criterion, and with Gemini-generated chain-of-thought rationales placed inside the FIM middle span. The authors mid-train Qwen2.5-Coder-Instruct 7B/14B and Qwen3-8B, then apply existing agentic post-training pipelines (R2E-Gym, SWE-Smith, SWE-Lego). They report consistent SWE-Bench-Verified/Lite gains of +2.8/+3.0/+3.2 and +3.7/+4.0/+5.4 across models, plus reduced capability erosion on LiveCodeBench, BFCL, and τ-bench. The paper interprets these gains as evidence for a structural isomorphism between function call sites and agent action–observation–continuation loops.

Significance. If the central claim holds, this is a practically valuable result: a self-supervised, corpus-only stage on ordinary Python functions improves downstream coding-agent performance across model sizes and post-training pipelines, while also mitigating post-training capability erosion. The paper has real strengths: it openly releases the corpus, selection pipeline, and checkpoints; it provides detailed algorithmic appendices; it includes ablations separating rationale source, selection algorithm, and mask granularity; and it states its limitations explicitly in Section 6. The consistent directional gains across multiple configurations are encouraging. However, the two load-bearing issues below -- the absence of a matched generic-continued-pretraining control and the fragility of the 7B headline against the official baseline -- mean the current evidence does not yet establish that the function-aware structure, rather than generic exposure to 2.6B Python tokens, is the cause of the gains.

major comments (3)
  1. [§3.4, Table 3] The central causal claim -- that function-aware FIM structure drives the gains -- is not tested against the most natural confound: generic continued pretraining on the same corpus. Every row in Table 3 uses a FIM-formatted objective on the same Python corpus, with only selection or CoT source varied. In particular, the 'FIM, no CoT' row already improves the average by +1.18 over baseline, which is roughly half of the full Gemini-CoT gain and is fully compatible with an explanation based on exposure to 2.6B tokens of ordinary Python. A matched-budget condition with an ordinary left-to-right (or random-span) objective on the same corpus is needed to attribute the effect to function-aware FIM. Without it, the §3.3 statement that cross-domain recovery on τ-bench and BFCL is 'direct evidence for the function-call/tool-call isomorphism' rests on an untested contrast.
  2. [§3.2, Table 1] The flagship 7B result is measured against a reproduced R2E-Gym baseline of 15.00, while the official R2E-Gym number reported in the same table is 19.00. The mid-trained+post-trained checkpoint achieves 17.80, which is below the official baseline. Thus the abstract's '+2.8' gain is not robust to which baseline is used; it is only a gain relative to the authors' reproduction. The 14B result is less affected (29.20 vs. official 26.80), but the 7B claim is the one emphasized in the abstract. The paper should either reconcile the 15.00 vs. 19.00 reproduction gap or report the delta against the official baseline prominently. Additionally, the 7B comparison has overlapping standard-deviation bands (17.80±1.40 vs. 15.00±1.50) with only three seeds, so statistical significance is not established.
  3. [§3.4, Table 3] The ablation table reports no uncertainty estimates or seed counts, yet the differences between adjacent selection variants are small -- e.g., 'PDG + Î' vs. 'Full' differ by 0.30 on Verified (16.70 vs. 17.00) and 0.25 on average (15.35 vs. 15.60), while the main table's reported standard deviations are on the order of 1.0–1.5 points. The claim that 'the function-selection algorithm is the dominant lever' and that 'Ĥ and Î are not redundant' requires either multiple seeds or a formal significance test. As reported, these ablation deltas are within plausible noise.
minor comments (4)
  1. [Appendix B.6] The worked example for Calculator.total uses `self.history` in the code listing, while Figure 2 uses `self.hist` in the same method. The inconsistency should be fixed.
  2. [§3.4, Block (A)] The 'self-CoT' variant is underspecified: it is not clear whether the model under training generates only the rationale or also the candidate body. Since the body is normally Gemini-generated, this distinction matters for interpreting the 14.85 average.
  3. [Table 2] The 'Avg' column is a simple unweighted mean across six benchmarks with very different scales and baselines (e.g., Terminal-Bench 2.0 values are near 0–4, LiveCodeBench near 24–40). A simple mean can be driven by idiosyncrasies of one benchmark; consider reporting a normalized average or individual deltas more prominently.
  4. [§3.1, evaluation protocol] The paper says all numbers are means over three seeds, but Table 3 does not provide standard deviations or seed counts. Please state explicitly whether the ablations are single-run or multi-seed.

Circularity Check

0 steps flagged

No circularity found: the training objective and the evaluation benchmarks are distinct, selection hyperparameters are hand-designed rather than fitted to target scores, and the isomorphism is explicitly treated as an empirical hypothesis.

full rationale

The paper's central claim is an empirical comparison: base + post-training versus base + FIM mid-training + identical post-training, evaluated on external benchmarks (SWE-Bench, LiveCodeBench, tau-bench, BFCL). The mid-training objective masks functions selected by a hand-designed PDG-based complexity/inferability score; the paper states the components are 'hand-designed proxies' and the threshold is 'used throughout this work' without evidence of fitting to SWE-Bench scores. The ablation blocks in Table 3 hold the training budget fixed and vary only selection, CoT source, and mask granularity, so the comparisons do not reduce to the definition of the objective. The 'isomorphism' between agent steps and function call sites is explicitly presented as an analogy to test empirically ('an empirical question we address in Section 3') rather than as a derivation. The main empirical concern—that gains may reflect generic continued pretraining rather than function-aware FIM structure, given the absence of a matched left-to-right control—is a validity/causal-attribution issue, not a circularity: the prediction is not equivalent to the training input by construction. Likewise, the reproduced R2E-Gym baseline being below the official number weakens the headline delta but does not make any fitted value identical to the predicted outcome. No load-bearing self-citation, uniqueness theorem, or ansatz-via-citation was found.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The recipe rests on a large inventory of hand-set selection constants and a few domain assumptions. The free parameters are not fitted to benchmark outcomes, so the circularity burden is low, but the absence of sensitivity analysis means the method's advantage could be contingent on the specific hand-tuned operating point. The most fragile assumptions are the isomorphism claim and the faithfulness of reproduced baselines.

free parameters (6)
  • FIM selection threshold τ_FIM = 0.08 (single), 0.04 (pairs), 0.03 (triples)
    Hand-set admission threshold for masking targets (Eq. 3, Algorithm 1, Appendices B.5/B.7); no sensitivity analysis reported.
  • Complexity-score weights and caps = (wℓ,wc,wd)=(0.4,0.4,0.2); (cℓ,cc,cd)=(50,10,5)
    Constants in Eq. (1) chosen to match corpus medians; they directly control which functions are considered complex enough to mask.
  • Inferability-score weights = (α,β,γ,δ,ε)=(0.30,0.25,0.20,0.10,0.15)
    Mixing weights in Eq. (2), plus roughly 15 micro-constants in Table 6 (call-site bonuses, normalization constants); all hand-set.
  • Difficulty penalty parameters = τd=0.50, σ=0.20 (single); τd_G=0.55, σG=0.20 (groups)
    One-sided Gaussian damping in Eq. (5) determines which high-difficulty targets are downweighted.
  • Group-selection thresholds and caps = τ_coup=0.15, θ2=0.30, θ3=0.40, N2=5, N3=3
    Algorithm 2 criteria governing the 80%/15%/5% single/pair/triple mixture used in the main recipe.
  • CoT filter cutoff and discard rule = keep top ~400K; discard infeasible or executability=1 or (overall≤2 and executability≤2)
    Gemini judge thresholds in Appendix B.10 determine which rationales and predicted bodies enter the training target.
axioms (5)
  • domain assumption Python AST parsing and short-name call resolution recover the true function dependency structure.
    All target selection depends on PDG edges E_call and E_sib (Section 2.3.1, B.2); incorrect resolution biases which functions are masked.
  • domain assumption Function call sites and coding-agent steps are structurally isomorphic, and training on the former transfers to the latter.
    Section 2.1 states this as the motivating isomorphism; if false, the cross-domain gains are attributable to generic code pretraining rather than an agent-relevant inductive bias.
  • domain assumption Gemini-3-Flash rationales and predicted bodies, filtered by an LLM judge, are a reliable training signal.
    The default recipe trains on 400K Gemini-generated CoT-body pairs (Appendices B.9-B.10) with no human verification of the generated implementations.
  • domain assumption The reproduced post-training baselines are faithful to the official pipelines.
    Table 1 shows reproduced R2E-Gym Verified at 15.00 versus the officially reported 19.00 on 7B, a 4-point gap; all headline deltas are computed against the reproduced baselines.
  • domain assumption The mid-training corpus is not contaminated with SWE-Bench test content.
    Decontamination is by repository name/fork removal and commit timestamps (Section 2.2); this cannot fully rule out content copied into other repositories or benchmark instances.

pith-pipeline@v1.3.0-alltime-deepseek · 23870 in / 13026 out tokens · 130212 ms · 2026-08-02T06:28:59.858762+00:00 · methodology

0 comments
read the original abstract

Coding agents must integrate external tool returns into ongoing reasoning - a capability that standard left-to-right pretraining on code exposes only in its forward direction. We observe that the action-observation-continuation loop of a coding agent is structurally isomorphic to a function call site, where a caller binds arguments, a callee returns a value computed elsewhere, and downstream code consumes that value. This conditioning structure exists at internet scale in ordinary code. We exploit it through function-aware fill-in-the-middle (FIM) mid-training: a self-supervised objective that masks functions selected via program dependency graph analysis and a complexity-inferability double criterion. We mid-train Qwen2.5-Coder-Instruct (7B/14B) and Qwen3-8B on a 2.6B-token decontaminated corpus drawn from 968 GitHub repositories, then apply existing agentic post-training pipelines. Mid-training improves SWE-Bench-Verified by +2.8/+3.0 at 7B/14B and by +3.2 on Qwen3-8B; SWE-Bench-Lite gains are +3.7/+4.0/+5.4 on the same models. The improvement holds across two post-training pipelines (R2E-Gym, SWE-Smith) and on a non-Qwen2.5 base (Qwen3-8B with SWE-Lego). Beyond in-domain gains, mid-training also mitigates the capability erosion that agentic post-training otherwise inflicts on non-agent coding (e.g., LiveCodeBench) and non-coding tool-use benchmarks (tau-bench, BFCL): although the mid-training corpus contains Python code only, the function-call inductive bias survives post-training and yields consistent gains.

Figures

Figures reproduced from arXiv: 2607.12463 by Cong Wei, Jiarong Liang, Ping Nie, Wenhu Chen, Xuye Liu, Yubo Wang, Yuxuan Zhang, Yuyu Zhang.

Figure 1
Figure 1. Figure 1: Left: A function call site and a single step of a coding agent are structurally similar, decomposing into the same four stages: context, call/action, return/observation, continuation. Middle: We exploit this analogy via function-aware FIM mid-training. A function B is selected from the program dependency graph using complexity (Hˆ ) and inferability (ˆI) scores; the model is then mid-trained to fill in B’s… view at source ↗
Figure 2
Figure 2. Figure 2: Function-aware FIM target selection on a small calculator example. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of the 968 source repositories across ten topic categories. The corpus is dominated by reference implementations, scientific computing, and small frameworks; compiler and networking/security tails are kept by design to maintain coverage diversity [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: License distribution of the 968-repository corpus. Permissive licenses (MIT, Apache-2.0, BSD) account for over 80% of the corpus. Small categories (LGPL, ISC, Boost, MIT-0, Unlicense, CC0, CC BY, CC BY-NC, etc.) are aggregated as “Other research-permissive licenses,” all of which permit at least non-commercial research use. Algorithm 1 Single-function FIM target selection for one file. Require: Source file… view at source ↗
Figure 5
Figure 5. Figure 5: reports the pass-rate stratification used in Section 4.2: the largest gain (+9.1 pp) is on multi-function single-file tasks (n=88), more than 4× the +2.1 pp gain on single-function tasks (n=341); multi-file tasks (n=71) remain hard for both checkpoints (∼11.3% each). single-func single-file multi-func single-file multi-file 0 10 20 30 40 Pass rate (%) 32.5 34.6 n=341 13.6 22.7 n=88 11.3 11.3 n=71 Pass rate… view at source ↗
Figure 6
Figure 6. Figure 6: Outcome distribution per evaluation run on SWE-Bench-Verified (14B, R2E-Gym), aver [PITH_FULL_IMAGE:figures/full_fig_p023_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Pass rate on SWE-Bench-Lite stratified by gold-patch shape, averaged over three evaluation [PITH_FULL_IMAGE:figures/full_fig_p024_7.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 13 linked inside Pith

  1. [1]

    Front-loading reasoning: The synergy between pretraining and post-training data.arXiv preprint arXiv:2510.03264, 2025

    Syeda Nahida Akter, Shrimai Prabhumoye, Eric Nyberg, Mostofa Patwary, Mohammad Shoeybi, Yejin Choi, and Bryan Catanzaro. Front-loading reasoning: The synergy between pretraining and post-training data.arXiv preprint arXiv:2510.03264, 2025

    arXiv 2025

  2. [2]

    Efficient training of language models to fill in the middle, 2022

    Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle, 2022

    2022

  3. [3]

    Unveiling the key factors for distilling chain-of- thought reasoning

    Xinghao Chen, Zhijing Sun, Guo Wenjin, Miaoran Zhang, Yanjun Chen, Yirong Sun, Hui Su, Yijie Pan, Dietrich Klakow, Wenjie Li, et al. Unveiling the key factors for distilling chain-of- thought reasoning. InFindings of the Association for Computational Linguistics: ACL 2025, pages 15094–15119, 2025

    2025

  4. [4]

    Dataflow-guided retrieval augmentation for repository- level code completion

    Wei Cheng, Yuhan Wu, and Wei Hu. Dataflow-guided retrieval augmentation for repository- level code completion. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7957–7977, 2024

    2024

  5. [5]

    Fullstack bench: Evaluating llms as full stack coders, 2024

    Yao Cheng, Jianfeng Chen, Jie Chen, Li Chen, Liyu Chen, Wentao Chen, Zhengyu Chen, Shijie Geng, Aoyan Li, Bo Li, et al. Fullstack bench: Evaluating llms as full stack coders, 2024

    2024

  6. [6]

    Swe-bench pro: Can ai agents solve long-horizon software engineering tasks?arXiv preprint arXiv:2509.16941, 2025

    Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, et al. Swe-bench pro: Can ai agents solve long-horizon software engineering tasks?arXiv preprint arXiv:2509.16941, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Horizon- length prediction: Advancing fill-in-the-middle capabilities for code generation with lookahead planning

    Yifeng Ding, Hantian Ding, Shiqi Wang, Qing Sun, Varun Kumar, and Zijian Wang. Horizon- length prediction: Advancing fill-in-the-middle capabilities for code generation with lookahead planning. 2024

    2024

  8. [8]

    Incoder: A generative model for code infilling and synthesis.arXiv preprint arXiv:2204.05999, 2022

    Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis.arXiv preprint arXiv:2204.05999, 2022

    Pith/arXiv arXiv 2022

  9. [9]

    Distil-whisper: Robust knowledge distillation via large-scale pseudo labelling, 2023

    Sanchit Gandhi, Patrick V on Platen, and Alexander M Rush. Distil-whisper: Robust knowledge distillation via large-scale pseudo labelling, 2023

    2023

  10. [10]

    Training long-context, multi-turn software engineering agents with reinforcement learning.arXiv preprint arXiv:2508.03501, 2025

    Alexander Golubev, Maria Trofimova, Sergei Polezhaev, Ibragim Badertdinov, Maksim Nekra- shevich, Anton Shevtsov, Simon Karasik, Sergey Abramov, Andrei Andriushchenko, Filipp Fisin, et al. Training long-context, multi-turn software engineering agents with reinforcement learning.arXiv preprint arXiv:2508.03501, 2025

    arXiv 2025

  11. [11]

    Structure-aware fill-in-the- middle pretraining for code.arXiv preprint arXiv:2506.00204, 2025

    Linyuan Gong, Alvin Cheung, Mostafa Elhoushi, and Sida Wang. Structure-aware fill-in-the- middle pretraining for code.arXiv preprint arXiv:2506.00204, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    Ast-t5: Structure-aware pretraining for code generation and understanding.arXiv preprint arXiv:2401.03003, 2024

    Linyuan Gong, Mostafa Elhoushi, and Alvin Cheung. Ast-t5: Structure-aware pretraining for code generation and understanding.arXiv preprint arXiv:2401.03003, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Olmo: Accelerating the science of language models

    Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15789–15809, 2024

    2024

  14. [14]

    DeepSeek-Coder: When the large language model meets programming – the rise of code intelligence, 2024

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yifan Wu, YK Li, et al. DeepSeek-Coder: When the large language model meets programming – the rise of code intelligence, 2024

    2024

  15. [15]

    Don’t stop pretraining: Adapt language models to domains and tasks

    Suchin Gururangan, Ana Marasovi´c, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 8342–8360, 2020

    2020

  16. [16]

    Large language models are reasoning teachers

    Namgyu Ho, Laura Schmid, and Se-Young Yun. Large language models are reasoning teachers. InProceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), pages 14852–14882, 2023. 10

    2023

  17. [17]

    Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes

    Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. InFindings of the Association for Computational Linguistics: ACL 2023, pages 8003–8017, 2023

    2023

  18. [18]

    Minicpm: Unveiling the potential of small language models with scalable training strategies, 2024

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies, 2024

    2024

  19. [19]

    Remit: Rl-guided mid-training for iterative llm evolution.arXiv preprint arXiv:2602.03075, 2026

    Junjie Huang, Jiarui Qin, Di Yin, Weiwen Liu, Yong Yu, Xing Sun, and Weinan Zhang. Remit: Rl-guided mid-training for iterative llm evolution.arXiv preprint arXiv:2602.03075, 2026

    arXiv 2026

  20. [20]

    Qwen2.5-Coder technical report, 2024

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2.5-Coder technical report, 2024

    2024

  21. [21]

    Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024

    2024

  22. [22]

    R2E- Gym: Procedural environments and hybrid verifiers for scaling open-weights SWE agents, 2025

    Naman Jain, Jaskirat Singh, Manish Shetty, Liang Zheng, Koushik Sen, and Ion Stoica. R2E- Gym: Procedural environments and hybrid verifiers for scaling open-weights SWE agents, 2025

    2025

  23. [23]

    SWE-bench: Can language models resolve real-world GitHub issues? InThe twelfth international conference on learning representations, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? InThe twelfth international conference on learning representations, 2023

    2023

  24. [24]

    StarCoder: May the source be with you!, 2023

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. StarCoder: May the source be with you!, 2023

    2023

  25. [25]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    Graphcoder: Enhancing repository-level code completion via code context graph-based retrieval and language model.arXiv preprint arXiv:2406.07003, 2024

    Wei Liu, Ailun Yu, Daoguang Zan, Bo Shen, Wei Zhang, Haiyan Zhao, Zhi Jin, and Qianxiang Wang. Graphcoder: Enhancing repository-level code completion via code context graph-based retrieval and language model.arXiv preprint arXiv:2406.07003, 2024

    Pith/arXiv arXiv 2024

  27. [27]

    Starcoder 2 and the stack v2: The next generation.arXiv preprint arXiv:2402.19173, 2024

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Noua- mane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation.arXiv preprint arXiv:2402.19173, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces, 2026

    Mike A Merrill, Alexander G Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E Kelly Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces, 2026

    2026

  29. [29]

    Orca: Progressive learning from complex explanation traces of gpt-4, 2023

    Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4, 2023

    2023

  30. [30]

    Codegen: An open large language model for code with multi-turn program synthesis.arXiv preprint arXiv:2203.13474, 2022

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis.arXiv preprint arXiv:2203.13474, 2022

    Pith/arXiv arXiv 2022

  31. [31]

    Training software engineering agents and verifiers with SWE-Gym, 2024

    Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with SWE-Gym, 2024

    2024

  32. [32]

    The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models

    Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. InForty-second International Conference on Machine Learning, 2025

    2025

  33. [33]

    Code Llama: Open foundation models for code, 2023

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code Llama: Open foundation models for code, 2023. 11

    2023

  34. [34]

    Bridging developer instructions and code completion through instruction-aware fill-in-the- middle paradigm.arXiv preprint arXiv:2509.24637, 2025

    Zhensu Sun, Chengran Yang, Chao Peng, Pengfei Gao, Xiaoning Du, Li Li, and David Lo. Bridging developer instructions and code completion through instruction-aware fill-in-the- middle paradigm.arXiv preprint arXiv:2509.24637, 2025

    arXiv 2025

  35. [35]

    SWE-Lego: Pushing the limits of supervised fine-tuning for software issue resolving, 2026

    Chaofan Tao, Jierun Chen, Yuxin Jiang, Kaiqi Kou, Shaowei Wang, Ruoyu Wang, Xiaohui Li, Sidi Yang, Yiming Du, Jianbo Dai, et al. SWE-Lego: Pushing the limits of supervised fine-tuning for software issue resolving, 2026

    2026

  36. [36]

    A survey on llm mid-training.arXiv preprint arXiv:2510.23081, 2025

    Chengying Tu, Xuemiao Zhang, Rongxiang Weng, Rumei Li, Chen Zhang, Yang Bai, Hongfei Yan, Jingang Wang, and Xunliang Cai. A survey on llm mid-training.arXiv preprint arXiv:2510.23081, 2025

    arXiv 2025

  37. [37]

    OpenHands: An open platform for AI software developers as generalist agents

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. OpenHands: An open platform for AI software developers as generalist agents. 2024

    2024

  38. [38]

    Ojbench: A competition level code benchmark for large language models, 2025

    Zhexu Wang, Yiping Liu, Yejie Wang, Wenyang He, Bofei Gao, Muxi Diao, Yanxu Chen, Kelin Fu, Flood Sung, Zhilin Yang, et al. Ojbench: A competition level code benchmark for large language models, 2025

    2025

  39. [39]

    Toward training superintelligent software agents through self-play swe-rl.arXiv preprint arXiv:2512.18552, 2025

    Yuxiang Wei, Zhiqing Sun, Emily McMilin, Jonas Gehring, David Zhang, Gabriel Synnaeve, Daniel Fried, Lingming Zhang, and Sida Wang. Toward training superintelligent software agents through self-play swe-rl.arXiv preprint arXiv:2512.18552, 2025

    Pith/arXiv arXiv 2025

  40. [40]

    Magicoder: Empow- ering code generation with oss-instruct.arXiv preprint arXiv:2312.02120, 2023

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empow- ering code generation with oss-instruct.arXiv preprint arXiv:2312.02120, 2023

    Pith/arXiv arXiv 2023

  41. [41]

    Agentless: Demystifying llm-based software engineering agents.arXiv preprint arXiv:2407.01489, 2024

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llm-based software engineering agents.arXiv preprint arXiv:2407.01489, 2024

    Pith/arXiv arXiv 2024

  42. [42]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report, 2025

    2025

  43. [43]

    SWE-agent: Agent–computer interfaces enable automated software engineering

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent–computer interfaces enable automated software engineering. volume 37, pages 50528–50652, 2024

    2024

  44. [44]

    SWE-smith: Scaling data for software engineering agents, 2025

    John Yang, Kilian Lieret, Carlos E Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. SWE-smith: Scaling data for software engineering agents, 2025

    2025

  45. [45]

    τ-bench: A benchmark for tool-agent-user interaction in real-world domains, 2024

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains, 2024

    2024

  46. [46]

    Multi-swe-bench: A multilingual benchmark for issue resolving.arXiv preprint arXiv:2504.02605, 2025

    Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, et al. Multi-swe-bench: A multilingual benchmark for issue resolving.arXiv preprint arXiv:2504.02605, 2025

    Pith/arXiv arXiv 2025

  47. [47]

    Other research-permissive licenses

    Liang Zeng, Yongcong Li, Yuzhen Xiao, Changshi Li, Chris Yuhao Liu, Rui Yan, Tianwen Wei, Jujie He, Xuchen Song, Yang Liu, et al. Skywork-swe: Unveiling data scaling laws for software engineering in llms.arXiv preprint arXiv:2506.19290, 2025. A Corpus Details This appendix expands on the data-collection summary (Section 2.2). We report category coverage, ...

    Pith/arXiv arXiv 2025