Pith. sign in

REVIEW 3 major objections 6 minor 22 references

Distilling full-cycle, source-free program rebuild trajectories lifts a 27B model into the performance band of much larger coding systems.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 12:08 UTC pith:3FCYCEVC

load-bearing objection Real systems contribution and solid gains, but the 5.7× token burn without a budget-matched baseline leaves the “whole-life-cycle skill” story underdetermined. the 3 major comments →

arxiv 2607.27146 v1 pith:3FCYCEVC submitted 2026-07-29 cs.SE cs.CLcs.LG

MindForge: Teaching Small Language Models Whole-Life-Cycle Software Engineering via Source-Free Program Synthesis

Yihao Chen , Shi Chang , Khaled Chawa , Feng Lin , Boyuan Chen , Shaowei Wang , Ahmed E. Hassan This is my paper
classification cs.SE cs.CLcs.LG
keywords source-free program synthesiscoding agentswhole-life-cycle software engineeringtrajectory distillationProgramBenchsupervised fine-tuninglong-horizon agentsCLI environments
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Building a program entirely from scratch—inferring the spec, designing architecture, implementing, debugging, testing, and refining—is far harder for coding agents than editing an existing codebase, and even frontier models fully solve under 1% of ProgramBench tasks. The bottleneck is training data: prior environment pipelines mostly teach single-phase edits on visible source. MindForge turns open-source command-line programs into source-free sandboxes that expose only a compiled reference binary and its docs, then collects long teacher trajectories that span the whole development life cycle. Fine-tuning a 27B model on about a thousand refined trajectories raises its ProgramBench average test pass rate from roughly 38% to 50% and produces consistent gains on seven held-out software-engineering benchmarks, from repository generation and translation to bug fixing and feature work. The practical claim is that whole-life-cycle, source-free supervision transfers real long-horizon engineering behavior into a small open model.

Core claim

A scalable pipeline that converts open-source CLIs into source-free environments, collects and refines full-life-cycle program-synthesis trajectories from a strong teacher, and distills them into Qwen3.6-27B raises ProgramBench average test pass rate from 37.98% to 49.51% and yields statistically significant gains on all seven unseen software-engineering benchmarks.

What carries the argument

MindForge: an automated pipeline that builds cleanroom Docker environments (reference executable plus sanitized docs only), rolls out teacher agents inside them, then applies infrastructure-noise recovery and selective reasoning rewrite so only coherent, buildable whole-life-cycle trajectories become supervised fine-tuning data.

Load-bearing premise

Success-filtered teacher runs on source-free CLI rebuilds—kept when they produce a buildable submission, without test-suite rejection sampling, and after host-side recovery plus selective reasoning edits—are clean enough that the measured gains reflect transferable whole-life-cycle skill rather than scaffold habits or leftover teacher artifacts.

What would settle it

Train the same 27B base on an equal volume of short single-phase issue-fix trajectories under the identical scaffold and context budget; if ProgramBench and the held-out suite gains disappear or reverse, the whole-life-cycle source-free recipe is not doing the claimed work.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Small open models can reach near-frontier ProgramBench partial-credit scores without matching frontier parameter counts when given full-cycle rebuild supervision.
  • Source-free CLI environments become a reusable training substrate for long-horizon agents, not only an evaluation setting.
  • Gains transfer beyond from-scratch rebuild to bug fixing, feature implementation, multilingual issue resolution, and repository translation.
  • Behavioral signatures shift: longer productive trajectories, higher reference probing coverage, and higher rates of converting reasoning or failure recovery into code edits.
  • Releasing the pipeline, environments, trajectories, and 27B checkpoint enables others to scale or ablate whole-life-cycle distillation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If full-cycle rebuild data is the missing axis, mixing it with existing short-horizon issue corpora may compound rather than trade off against SWE-bench-style resolve rates.
  • The same cleanroom packaging could support reinforcement learning with coverage or differential-test rewards once the auxiliary coverage images are used, not only supervised imitation.
  • Failure modes that remain—hard instances still rarely fully resolved—may need multi-agent critique or explicit architecture search rather than more of the same single-agent rollouts.
  • Source-free training may reduce contamination risk in public benchmarks that currently leak from visible repository histories.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces MindForge, a pipeline that converts open-source CLI repositories into "source-free" training environments (compiled reference executable + documentation only), collects 1,001 whole-life-cycle program-synthesis trajectories from a GLM-5.2 teacher agent in 562 environments disjoint from ProgramBench, refines them via infrastructure-noise recovery and reasoning rewrites, and distills them into Qwen3.6-27B. The fine-tuned model improves from 37.98% to 49.51% average test pass rate on ProgramBench and shows statistically significant gains on all seven held-out SE benchmarks (RepoZero-C2Rust +31.00, DeepSWE +14.16, SWE-bench Verified +5.04, etc.). Behavioral analysis (turns, tool calls, failure rates, reasoning-to-edit transition rates, reference coverage) is offered as evidence that the gains reflect genuine whole-life-cycle skill transfer rather than superficial changes.

Significance. If the results hold, the paper makes two useful contributions: (1) a scalable, cross-language pipeline that converts open-source CLI programs into reproducible source-free environments (with behavior-equivalence replay and source-leakage checks), filling a real gap left by bug-fixing-centric frameworks like SWE-smith/SWE-Gym; and (2) evidence that a modest SFT recipe (973 trajectories, 27B student) yields broad transfer across eight evaluation settings spanning generation, translation, issue resolution, and feature implementation. Notable strengths: a documented contamination audit with per-instance verifier outcomes on overlapping repos (Table 8), paired statistical testing with Holm correction and bootstrap CIs across all nine comparisons (Table 9), a win/lose breakdown showing the gain is broad-based (152/43/5) rather than outlier-driven, and a promised release of environments, trajectories, pipeline, and model. The 5.7x inference-cost multiplier is disclosed openly, which is to the authors' credit, but it simultaneously weakens the causal story the title and §6 tell.

major comments (3)
  1. [§4.3, Table 4; §6] The paper's central interpretive claim — that gains reflect transfer of whole-life-cycle SE skill rather than learned persistence — is underdetermined because the fine-tuned model consumes 5.7x the base model's tokens (58.22M vs 10.13M mean per instance), 2.1x the turns, and 2.1x the tool calls, and no budget-matched control exists. Pass rate and agentic compute are known to be strongly coupled, and ProgramBench's partial-credit metric directly rewards the broader reference probing the fine-tuned model does more of (coverage 58.39% vs 49.34%). The supporting behavioral evidence (lower command-failure rate, doubled edit-after-reasoning/recovery transition rates) is suggestive but is itself correlated with horizon length, so it does not isolate the mechanism. A concrete, bounded test: evaluate the base model on a subset of ProgramBench instances under a matched turn/token budget (raisedcap
  2. [§2.2, Appendix B; contributions list] The refinement procedures are listed as a core contribution, but their effect is never isolated. There is no ablation comparing SFT on (a) raw success-filtered trajectories vs (b) recovered + rewritten ones, nor any quantification of how many of the 973 training trajectories were touched by each procedure. Similarly, the retention rule accepts any trajectory whose compile.sh builds, with no test-suite-based rejection sampling; the paper argues thresholds are unclear, but does not report the quality distribution (e.g., teacher-side pass rates) of retained trajectories. Without these, the claim that refinement yields 'clean supervision' responsible for the gains is asserted rather than shown. A small-scale ablation (e.g., 2-3 recipe variants on a data subset) would suffice.
  3. [§3.3, Appendix D.1] The headline number (37.98 -> 49.51) and the three other long-horizon benchmarks (NL2Repo, DeepSWE, RepoZero) rest on a single run per model. The task-level bootstrap CI [8.34, 14.74] captures instance sampling variance but not run-to-run variance of 700+-turn agentic rollouts, which in this literature is often several points. The win/lose split (152/43/5) mitigates this partially, but a second run on even a 50-instance subset would bound the scaffold-variance of the flagship result. This matters most for the smaller OOD gains (e.g., +4.56 NL2Repo-without-tests, +4.94 FeatBench) where the Holm-adjusted p=0.033 is already marginal.
minor comments (6)
  1. [Figure 1, Table 3] Figure 1 and Table 3 mix models evaluated by the authors under mini-swe-agent with numbers taken from the ProgramBench leaderboard and vendor technical reports, which may use different scaffolds, reasoning settings, and instance subsets. The caption notes this, but the 'comparable to substantially larger frontier models' framing in the abstract and §4.1 should carry the caveat inline, since cross-setting comparisons of agentic scores are fragile.
  2. [§4.2] The '9.0x increase' on DeepSWE is relative to a 1.76% base (roughly 2/113 tasks); framing a 14.16pp gain as a 9x multiplier overstates the effect. Recommend reporting percentage points primarily, as done elsewhere in the paper.
  3. [Table 4] Teacher (GLM-5.2) operational metrics are aggregated over the 1,001 training trajectories, while all other rows are on the 200 ProgramBench instances; these populations differ in difficulty mix and budget. The caption discloses this, but the text's claim that the student's tool-call volume 'exceeds even the teacher's' should be tempered accordingly.
  4. [§1, Table 4] The 830-turn, 209.5M-token single run cited in the introduction as evidence of 'sustained, productive persistence' needs context: how it fits the stated 512K evaluation context window, and whether such outliers are included in the Table 4 means. As stated it reads as anecdote.
  5. [§2.3] TypeScript is counted among the 'six compiled programming languages' (1 instance, 0.2%); either justify or reclassify. Relatedly, 'small language model' for a 27B dense model is relative framing worth one clarifying phrase.
  6. [§2.3, Table 2, Appendix C.1] Appendix C.1's honesty about the rule-based activity mining (proxies, fixed 10/15-event windows, no precision/recall validation) is commendable, but Table 2's headline coverage numbers (99.1%, 87.1%, etc.) are presented in §2.3 without these caveats. A forward pointer to the limitations would prevent overreading.

Circularity Check

0 steps flagged

No derivation-chain circularity: gains are measured on external hidden tests and held-out third-party benchmarks, not quantities defined by the training objective.

full rationale

MindForge is an empirical systems/SFT paper, not a first-principles derivation. The load-bearing claim is that distilling 1,001 success-filtered whole-life-cycle trajectories raises Qwen3.6-27B’s ProgramBench average test pass rate (37.98%→49.51%) and improves seven unseen SE benchmarks. Training environments are built from repositories the paper states are disjoint from ProgramBench; evaluation disables internet; OOD suites (SWE-bench*, RepoZero, NL2Repo, DeepSWE, FeatBench) use different task formulations and external verifiers. Pass-rate and resolve-rate metrics are not algebraic rearrangements of the SFT loss, the success filter (buildable compile.sh), or the GLM-5.2 rewrite step. Self-use of GLM-5.2 as teacher and repair model is methodological consistency, not a uniqueness theorem or fitted parameter renamed as prediction. Residual concerns (inference-budget confounding, scaffold habits) are causal-attribution risks outside this circularity rubric. No self-definitional loop, fitted-input-as-prediction, load-bearing self-citation uniqueness claim, or renamed known identity was found.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 1 invented entities

This is an empirical systems paper. Load-bearing premises are domain choices about task design, data filtering, and evaluation protocol rather than fitted physical constants or new ontological entities. The claim rests on standard SFT practice plus the assumption that cleanroom CLI rebuild trajectories are high-value whole-life-cycle supervision.

free parameters (3)
  • SFT hyperparameters (lr peak 4e-5, 8 epochs, global batch 96, AdamW betas/wd, seed 1105, 256k length filter) = lr 4e-5→4e-6 cosine; 8 epochs; seed 1105
    Chosen training recipe that produces the reported checkpoint; not swept exhaustively against the claim in the main text.
  • Trajectory retention rule (completion command + successful compile.sh; no test-suite reject sampling) = 1,001 complete / 973 used after 256k filter
    Defines which teacher rollouts enter SFT; directly shapes the data distribution underlying all reported gains.
  • Activity-mining temporal windows (10 / 15 tool events) and lexical design patterns = 10 events localize/fix; 15 events refine
    Rule thresholds used for Table 2 coverage claims about multi-stage signal; acknowledged as operational proxies.
axioms (5)
  • domain assumption Source-free CLI reimplementation from a reference binary plus docs is a valid, transferable proxy for whole-life-cycle software engineering supervision.
    Core framing in Introduction and Section 2; training task differs from several OOD benchmarks yet transfer is interpreted as life-cycle skill.
  • domain assumption Buildable submission without hidden-test rejection sampling is sufficient quality control for distillation targets.
    Section 2.2 explicitly avoids test-suite reject sampling due to unclear success thresholds.
  • ad hoc to paper Infrastructure replay plus local reasoning rewrite preserves faithful teacher behavior for supervised learning.
    Appendix B procedures alter trajectory text while freezing tool calls; safety screen is model-judged.
  • domain assumption Mini-SWE-Agent scaffold and disabled-internet evaluation faithfully measure agent capability rather than harness artifacts.
    Section 3.3 evaluation protocol shared across base and fine-tuned models.
  • domain assumption Standard supervised fine-tuning with assistant-token loss masking can transfer long-horizon agent policies.
    Section 3.2 training configuration; no RL in the main recipe.
invented entities (1)
  • MindForge cleanroom environment + trajectory-refinement pipeline independent evidence
    purpose: Scalable source-free whole-life-cycle training data generation for coding agents.
    Methodological construct, not a physical entity; evidence is empirical performance and promised artifact release rather than an external conserved quantity.

pith-pipeline@v1.2.0-grok45-kimik3 · 34498 in / 3460 out tokens · 77009 ms · 2026-07-30T12:08:29.981961+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of MindForge: Teaching Small Language Models Whole-Life-Cycle Software Engineering via Source-Free Program Synthesis." pith.science (2026). https://pith.science/paper/3FCYCEVC

@misc{pith2026260727146,
  author       = {Pith},
  title        = {Pith review of: MindForge: Teaching Small Language Models Whole-Life-Cycle Software Engineering via Source-Free Program Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3FCYCEVC}},
  note         = {Machine review of arXiv:2607.27146}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Coding agents have made substantial progress on software engineering tasks that modify existing codebases, including bug fixing and feature implementation. However, constructing a complete program from scratch remains a major challenge: even the frontier models evaluated on ProgramBench fully resolve fewer than 1% of tasks. One obstacle is the lack of scalable training environments for this from-scratch setting, spanning the whole software engineering life cycle, as existing environment-construction frameworks focus only on a single phase in software development. To address this gap, we introduce MindForge, an automated pipeline that converts open-source command-line programs into source-free environments that expose only a compiled reference executable and its documentation. Using MindForge, we construct training environments from repositories disjoint from those in ProgramBench, and curate a high-quality data recipe consisting of program synthesis trajectories using GLM-5.2 as the teacher agent. Fine-tuning Qwen3.6-27B on these trajectories increases its ProgramBench average test pass rate from 37.98% to 49.51%, achieving performance comparable to substantially larger frontier models. Moreover, the fine-tuned model consistently improves over the base model across all seven unseen software engineering benchmarks, spanning long-horizon repository generation and translation, bug fixing, feature implementation, and cross-language issue resolution, with absolute gains of 31.00 points on RepoZero-C2Rust, 14.16 on DeepSWE, 10.70/4.56 on NL2Repo-Bench (with/without tests), 5.04 on SWE-bench Verified, 5.93 on SWE-bench Pro, 5.22 on SWE-bench Multilingual, and 4.94 on FeatBench.

Figures

Figures reproduced from arXiv: 2607.27146 by Ahmed E. Hassan, Boyuan Chen, Feng Lin, Khaled Chawa, Shaowei Wang, Shi Chang, Yihao Chen.

Figure 1
Figure 1. Figure 1: Average ProgramBench test pass rate by model scale. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Generalization to seven software-engineering [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Illustrative infrastructure-noise recovery. After a transient interruption, the host rewinds to the last healthy step, [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Reasoning rewrite after malformed tool use. Cleanup removes the malformed turn and its scaffold error, the retained [PITH_FULL_IMAGE:figures/full_fig_p020_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: An example of four manually examined trajectories that excercise different software engineering life-cycle activities [PITH_FULL_IMAGE:figures/full_fig_p022_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

22 extracted references · 9 linked inside Pith

  1. [2]

    We find no repository-identity overlap for RepoZero C2Rust, SWE-bench Verified, SWE-bench Pro, or FeatBench

    Five repository identities overlap, covering 17 evaluation instances: three DeepSWE instances and 14 SWE-bench Multilingual instances. We find no repository-identity overlap for RepoZero C2Rust, SWE-bench Verified, SWE-bench Pro, or FeatBench. NL2Repo’s public metadata exposes target package names rather than GitHubrepository identities;comparing thesepac...

    2026

  2. [3]

    arXiv preprint arXiv:2509.22237

    FeatBench: Evaluating CodingAgentsonFeatureImplementationforVibeCoding. arXiv preprint arXiv:2509.22237. Chen, S.; Hafeez, A.; and Paracha, S

    arXiv

  3. [4]

    Chowdhury, N.; Aung, J.; Chan, J

    Signals: Tra- jectory Sampling and Triage for Agentic Interactions.arXiv preprint arXiv:2604.00356. Chowdhury, N.; Aung, J.; Chan, J. S.; Jaffe, O.; Sherburn, D.;Starace,G.;Mays,E.;Dias,R.;Aljubeh,M.;Glaese,M.; Jimenez,C.E.;Yang,J.;Ho,L.;Patwardhan,T.;Liu,K.;and Madry, A

    arXiv

  4. [6]

    Desai,R.;Hu,J.;Cabezas,J.;Harsola,N.;Shukla,P.;Wang, D.; Li, X.; Chaim, R

    Swe-bench pro: Can ai agents solve long-horizon software engineering tasks?arXiv preprint arXiv:2509.16941. Desai,R.;Hu,J.;Cabezas,J.;Harsola,N.;Shukla,P.;Wang, D.; Li, X.; Chaim, R. B.; Assadi, A. E.; Kamath, O. M.; Faldu,F.;Hebbar,P.;Sun,J.;Li,Y.;Srinivasan,P.;Gupta,I.; Settles,C.;Chen,D.;Raja,P.;Liu,A.;Šuppa,M.;Sasikumar, N.;Kong,L.;Quintanilla,E.;Berc...

    Pith/arXiv arXiv 2026

  5. [7]

    Facchinetti, T.; and others

    NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents.arXiv preprint arXiv:2512.12730. Facchinetti, T.; and others

    arXiv

  6. [8]

    Garrett-Harris, A.; Neidel, J.; and others

    davinci-env: Open swe environment synthesis at scale.arXiv preprint arXiv:2603.13023. Garrett-Harris, A.; Neidel, J.; and others

    arXiv

  7. [9]

    Jain, N.; Singh, J.; Shetty, M.; Zheng, L.; Sen, K.; and Sto- ica, I

    Deep- SWE: Measuring Frontier Coding Agents on Origi- nal, Long-Horizon Engineering Tasks.arXiv preprint arXiv:2607.07946. Jain, N.; Singh, J.; Shetty, M.; Zheng, L.; Sen, K.; and Sto- ica, I

    Pith/arXiv arXiv

  8. [10]

    Microsoft, arXiv:2509.16198

    RPG: A Repository Planning Graph for Unified and Scalable Codebase Generation. Microsoft, arXiv:2509.16198. Ma, Y.; Cao, R.; Cao, Y.; Zhang, Y.; Chen, J.; Liu, Y.; Liu, Y.; Li, B.; Huang, F.; and Li, Y

    arXiv

  9. [11]

    arXiv:2411.00622

    Lingma-SWE-GPT: AnOpenDevelopment-Process-CentricLanguageModelfor Automated Software Improvement. arXiv:2411.00622. Pan,J.;Wang,X.;Neubig,G.;Jaitly,N.;Ji,H.;Suhr,A.;and Zhang, Y

    Pith/arXiv arXiv

  10. [12]

    arXiv:2606.24855

    OpenThoughts-Agent: Data Recipes for Agentic Models. arXiv:2606.24855. Rastogi,A.;Yang,A.;Jiang,A.Q.;Liu,A.H.;Sablayrolles, A.; Héliou, A.; Martin, A.; Agarwal, A.; Ehrenberg, A.; Lo, A.; et al

    Pith/arXiv arXiv

  11. [14]

    Team, K.; et al

    Swe-lego: Pushing the limits of supervised fine-tuning for software issue resolving.arXiv preprint arXiv:2601.01426. Team, K.; et al

    arXiv

  12. [15]

    arXiv:2607.24653

    Kimi K3: Open Frontier Intelligence. arXiv:2607.24653. Wei, Y.; Duchenne, O.; Copet, J.; Carbonneaux, Q.; Zhang, L.; Fried, D.; Synnaeve, G.; Singh, R.; and Wang, S

    arXiv

  13. [16]

    Xie, C.; Li, B.; Gao, C.; Du, H.; Lam, W.; Zou, D.; and Chen,K.2025.SWE-Fixer:TrainingOpen-SourceLLMsfor EffectiveandEfficientGitHubIssueResolution

    Swe-rl:Advancingllmreasoningviareinforcementlearning on open software evolution. Xie, C.; Li, B.; Gao, C.; Du, H.; Lam, W.; Zou, D.; and Chen,K.2025.SWE-Fixer:TrainingOpen-SourceLLMsfor EffectiveandEfficientGitHubIssueResolution. InChe,W.; Nabende, J.; Shutova, E.; and Pilehvar, M. T., eds.,Findings oftheAssociationforComputationalLinguistics:ACL2025, 112...

    arXiv 2025

  14. [18]

    https: //z.ai/blog/glm-5.2

    GLM-5.2: Built for Long-Horizon Tasks. https: //z.ai/blog/glm-5.2. Accessed: 2026-07-16. Zan, D.; Huang, Z.; Liu, W.; Chen, H.; Xin, S.; Zhang, L.; Liu, Q.; Aoyan, L.; Chen, L.; Zhong, X.; et al

    2026

  15. [19]

    InPro- ceedings of the 62nd Annual Meeting of the Association for ComputationalLinguistics(Volume1:LongPapers),13643– 13658

    Zhang,K.;Li,J.;Li,G.;Shi,X.;andJin,Z.2024.Codeagent: Enhancing code generation with tool-integrated agent sys- tems for real-world repo-level coding challenges. InPro- ceedings of the 62nd Annual Meeting of the Association for ComputationalLinguistics(Volume1:LongPapers),13643– 13658. Zhang, Z.; Xu, Y.; Liang, J.; Li, W.; Chen, X.; Qian, L.; Pei, X.; Huan...

    2024

  16. [20]

    Zhao,J.;Chen,G.;Meng,F.;Li,M.;Chen,J.;Xu,H.;Sun,Y.; Zhao,W.X.;Song,R.;Zhang,Y.;etal.2026a

    RepoZero: Can LLMs Generate a Code Repository from Scratch? arXiv:2605.07122. Zhao,J.;Chen,G.;Meng,F.;Li,M.;Chen,J.;Xu,H.;Sun,Y.; Zhao,W.X.;Song,R.;Zhang,Y.;etal.2026a. Immersionin thegithubuniverse:Scalingcodingagentstomastery.arXiv preprint arXiv:2602.09892. Zhao, J.; Chen, G.; Meng, F.; Zhao, W. X.; Song, R.; Wen, J.-R.; and Jia, K. 2026b. DeNovoSWE: S...

    Pith/arXiv arXiv

  17. [21]

    severity

    SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning. arXiv:2408.05517. Zhou, Q.; Zhang, J.; Wang, H.; Hao, R.; Wang, J.; Han, M.; Yang,Y.;Wu,S.;Pan,F.;Fan,L.;etal.2026a. Featurebench: Benchmarking agentic coding for complex feature develop- ment.arXiv preprint arXiv:2602.10975. Zhou,Y.;Zheng,K.;Chen,Q.;Hu,M.;Sun,Q.;Xu,C.;and Chen, J. 2026b. OffSee...

    Pith/arXiv arXiv 2024

  18. [38]

    ProgramBench:CanLanguageModelsRebuild Programs From Scratch?arXiv preprint arXiv:2605.03546

    Yang, J.; Lieret, K.; Ma, J.; Thakkar, P.; Pedchenko, D.; Sootla, S.; McMilin, E.; Yin, P.; Hou, R.; Synnaeve, G.; etal.2026b. ProgramBench:CanLanguageModelsRebuild Programs From Scratch?arXiv preprint arXiv:2605.03546. Z.ai

    Pith/arXiv arXiv

  19. [2019]

    arXiv preprint arXiv:1909.08053

    Megatron-lm: Training multi- billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053. Tao, C.; Chen, J.; Jiang, Y.; Kou, K.; Wang, S.; Wang, R.; Li, X.; Yang, S.; Du, Y.; Dai, J.; et al

    Pith/arXiv arXiv 1909

  20. [2024]

    https: //openai.com/index/introducing-swe-bench-verified/

    Introducing SWE-bench Verified. https: //openai.com/index/introducing-swe-bench-verified/. Ope- nAI. Accessed: 2026-07-27. Copet, J.; Carbonneaux, Q.; Cohen, G.; Gehring, J.; Kahn, J.; Kossen, J.; Kreuk, F.; McMilin, E.; Meyer, M.; Wei, Y.; etal.2025. Cwm:Anopen-weightsllmforresearchoncode generation with world models. Deng,X.;Da,J.;Pan,E.;He,Y.Y.;Ide,C.;...

    2026

  21. [2025]

    Chen, H.; Li, C.; and Li, J

    Swe-rebench: An au- tomated pipeline for task collection and decontaminated evaluation of software engineering agents.arXiv preprint arXiv:2505.20411. Chen, H.; Li, C.; and Li, J

    arXiv

  22. [2026]

    Epoch AI, in collaboration with METR, arXiv:2606.30182

    MirrorCode: AI Can Rebuild Entire Programs From Behavior Alone. Epoch AI, in collaboration with METR, arXiv:2606.30182. Badertdinov, I.; Golubev, A.; Nekrashevich, M.; Shevtsov, A.; Karasik, S.; Andriushchenko, A.; Trofimova, M.; Litv- intseva, D.; and Yangel, B

    Pith/arXiv arXiv