Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Manner

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SWE-Flow synthesizes fully verifiable test-driven software engineering data from unit tests alone, and fine-tuning a 32B open code model on that data lifts it past most closed models on incremental coding tasks.

desk verdict A genuinely new pipeline for synthesizing incremental TDD data from unit tests, with honest limitations, but the 'fully verifiable' guarantee is asserted rather than demonstrated and the benchmark is self-referential. read the letter →

arxiv 2506.09003 v2 pith:U75YBUGW submitted 2025-06-10 cs.CL

Lei Zhang , Jiaxi Yang , Min Yang , Jian Yang , Mouxiang Chen , Jiajun Zhang , Zeyu Cui , Binyuan Hui
show 1 more author
Junyang Lin
This is my paper
classification cs.CL
keywords test-drivendevelopmentsoftwareengineeringdatasynthesisruntimedependencygraphunittestscodemodelfine-tuningincrementalverifiabletraininggenerationbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SWE-Flow claims that high-quality software engineering training data can be synthesized automatically from unit tests alone, without human-submitted issue reports. It executes a project's passing tests with a runtime hook, records the function-call relationships into a Runtime Dependency Graph, and derives a step-by-step development schedule where each step adds exactly the functions that a set of tests exercises. For every step it produces a skeletonized partial codebase, a requirement document generated from the test contents, and the reference diff back to the full code, making every task executable and verifiable by its unit tests. From 74 real GitHub projects it generates 16,061 training instances and a 2,020-task benchmark, and fine-tuning an open 32B code model on those instances improves its pass rate on the benchmark over the base model and over most closed models in both replace and patch formats.

What carries the argument

The Runtime Dependency Graph (RDG), built by SWE-Flow-Trace: a directed graph whose nodes are project functions invoked during unit test execution and whose edges are direct caller-callee pairs recorded from the function call stack, with calls outside the project filtered out. SWE-Flow-Schedule consumes the RDG, groups target test functions by the core functions they cover, sorts the groups by coverage size, and merges already-covered functions into earlier groups, producing a development schedule that respects dependency order. Skeletonization then turns each schedule step into a task: target core functions have their bodies removed and replaced by an LLM-generated docstring, while dependent core functions are deleted entirely, and the diff between the skeletonized and original codebases becomes the ground-truth solution.

What would settle it

Run SWE-Flow on a project that uses asyncio, multiprocessing, or callback-heavy patterns; for every scheduled step, apply the step's ground-truth patch to the skeletonized codebase and run that step's unit tests. If any step fails its own tests or fails to import because a needed function is missing, the RDG missed an edge and the verifiability claim is broken — the pattern the authors themselves flag in their limitations section.

Watch

Extended reading notes

Core claim

The central claim is that the incremental development history of a codebase can be reconstructed from its unit tests: each test is treated as a requirement, the functions it invokes form a Runtime Dependency Graph (RDG), and the topological order of that graph yields a natural development schedule. SWE-Flow-Trace captures the graph by monitoring the live function call stack while all passing tests run, filtering out calls outside the project; SWE-Flow-Schedule then merges tests that cover the same core functions and sorts them by the number of core functions involved, so each scheduled step rests only on functions developed in earlier steps. The paper maintains that every synthesized task is fully verifiable because correctness is decided by executing the step's unit tests in a containerized environment, and that this verifiability, together with automatic inference from real projects, removes the reliance on curated human issue reports. The empirical claim is that fine-tuning Qwen2.5-Coder-32B-Instruct on this data produces a model whose pass rates on SWE-Flow-Bench (Lite) improve on the base model and exceed those of most evaluated closed models in both the replace and patch solution formats.

Load-bearing premise

The load-bearing premise is that the Runtime Dependency Graph captured from the complete, passing test suite reflects every function interaction needed to rebuild the project incrementally, so that each scheduled step's partial codebase is buildable and its tests are passable.

Editorial extensions

If this is right

  • Fine-tuning open code models on SWE-Flow data improves their performance on unseen development tasks drawn from different projects, in both the replace and patch output formats.
  • Any GitHub project with an installable environment and a passing unit test suite becomes a source of verifiable training data, removing the dependence on human-submitted issues and manual filtering.
  • Because every task is verified by executing unit tests, the same pipeline can supply verifiable reward signals for reinforcement learning and large corpora for pre-training code models.
  • Merging consecutive schedule steps yields harder tasks, up to building an entire project from scratch, giving a configurable difficulty control for training and evaluation.
  • SWE-Flow-Bench grades incremental, test-guided development steps, complementing SWE-Bench's issue-patch evaluation and Commit0's one-shot full-library generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: part of the fine-tuning gain on SWE-Flow-Bench (Lite) could come from memorization, since the tasks are derived from real open-source code that the base model may already have seen; a contamination-controlled split using recently written or private projects would isolate the benefit of the data format itself.
  • Editorial inference: the runtime-tracing machinery is not Python-specific in principle, so porting the hook approach to other runtimes with call-stack inspection (e.g., a JavaScript engine inspector) would test whether the RDG-schedule recipe transfers to other ecosystems.
  • Editorial inference: the quality of a generated requirement document is bounded by the expressiveness of the unit test, so tests relying on mocking, fixtures, or external services may yield specifications that are under- or over-constrained relative to the intended function.
  • Editorial inference: a direct probe of the framework's core claim is to delete one dependency edge from the RDG before scheduling and check whether any step becomes unsolvable; the fraction of failures would quantify how much verifiability depends on graph completeness.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes SWE-Flow, a framework that synthesizes test-driven development (TDD) training and evaluation data from real-world GitHub repositories. The framework executes a project's unit tests under a Python tracing hook to build a Runtime Dependency Graph (RDG), derives a development schedule that orders function implementations by dependency size, and then skeletonizes the original codebase for each step by removing the implementations of target and dependent core functions. For each step it generates a requirement document, a partial codebase, and ground-truth patches in both replace and patch formats. The authors release 16,061 training instances and 2,020 test instances, introduce SWE-Flow-Bench, and fine-tune Qwen2.5-Coder-32B-Instruct on the synthesized data. They report that the fine-tuned model, SF-Coder-32B-Instruct, outperforms the base model and most closed models on SWE-Flow-Bench (Lite) in both replace and patch formats, and they claim that the synthesized data are fully verifiable because correctness is checked by executing unit tests.

Significance. If the central claims hold, SWE-Flow addresses a real bottleneck in software engineering data: the dependence on human-submitted issue reports. The idea of deriving incremental development steps from unit tests and using execution-based verification is timely and potentially scalable, and the public release of code, datasets, models, and Docker images is a concrete asset to the community. The paper also provides explicit algorithms (Algorithm 1 and Algorithm 2), which makes the method reproducible in principle. However, the current evidence does not fully support the 'fully verifiable' claim, the patch-format evaluation is weakened by a post-processing conversion, and the fine-tuning gains are measured only on a benchmark generated by the same pipeline as the training data. These issues are fixable with additional validation and clearer reporting, but they are load-bearing for the paper's main conclusions.

major comments (5)
  1. [Section 3.7 and Algorithm 1] The claim that every SWE-Flow task is 'fully verifiable' is not directly validated. The statement 'We only keep the unit tests that pass' (Section 3.7) refers to tests executed on the original full codebase, not to tests executed on each skeletonized codebase after applying the ground-truth patch. Because skeletonization removes entire function definitions (DCFNs) and replaces TCFN bodies with LLM-generated docstrings, the skeletonized codebase may fail to import or may otherwise not be re-executable even after the reference patch is applied. Please report, for every benchmark task, the result of applying the reference patch to the skeletonized codebase and running the step's unit tests, along with the number of tasks discarded due to import errors or undeclared dependencies. Without this measurement, the 'fully verifiable' property is an assumption rather than a demonstrated fact.
  2. [Section 4.1 and Appendix E.1] The 'Patch Format' results do not measure the quality of generated patches as patches. Appendix E.1 states that generated patches are 'almost entirely incompatible with system tools such as Linux's patch utility' and that the evaluation applies a post-processing step that converts generated patches into replace format before modifying the codebase. This means the patch-format pass rates in Table 5 and Figure 4 reflect the success of an internal repair/conversion process, not the model's ability to produce an applicable unified diff. Either evaluate patch-format outputs with an actual patch application tool that performs context matching, or clearly relabel the metric and report how often the conversion changes the outcome.
  3. [Section 5.2 and Section 4.1] The empirical validation of fine-tuning is circular in an important sense: SWE-Flow-Bench is generated by the same SWE-Flow pipeline that produced the training data. The test projects are disjoint from training projects, which mitigates direct memorization, but the task distribution, prompt format, skeletonization style, and requirement-document generation are identical. The reported improvement therefore demonstrates that the model learns the SWE-Flow task distribution, not that it improves general TDD-based coding ability. A comparison on an external benchmark such as SWE-Bench, Commit0, or a human-validated subset of real development tasks would substantially strengthen the claim that the synthesized data improve software engineering capability beyond the synthetic distribution.
  4. [Section 3.5] Skeletonization of Dependent Core Function Nodes removes both the function definition and its contents. A removed function that is referenced at module import time by code outside the current step (for example, as a decorator, a default argument, or a module-level assignment) will cause the skeletonized codebase to fail import, and no patch that only re-implements the current step's new functions can repair that. This is a concrete failure mode that is not captured by running tests on the original codebase and is not discussed in Section F.1, which lists only asyncio and multi-process limitations. The proposed per-task import-and-test validation in Section 3.7 would directly expose this issue.
  5. [Section F.1 and Table 3] The paper acknowledges that the RDG is incomplete for asynchronous and multi-process programs, but it never quantifies how often such incompleteness affects the synthesized tasks. Table 3 reports dependency depth and function counts, none of which measure missed-edge rates. Please provide a quantitative analysis of RDG completeness, for example by comparing the runtime graph against a static call graph (with appropriate caveats) or by reporting the number of steps whose tests fail on the skeletonized codebase even after the reference patch is applied. This is needed to assess the reliability of the development schedule and the 'fully verifiable' claim.
minor comments (5)
  1. [Section 5.2 and Figure 4] The model name is inconsistent: the text refers to SF-Coder-32B-Instruct, while Figure 4 uses both UF-Coder-32B-Instruct and UF-Coder-v4.0-iters-32. Please unify the naming across the paper.
  2. [Abstract and Section 3.7] The benchmark is called SWE-Flow-Eval in the abstract but SWE-Flow-Bench throughout the body. Please use one name consistently.
  3. [Table 3] The pylint Full Context Tokens value is listed as '23,76', which appears to be a typo for '2,376'.
  4. [Table 2] The marshmallow entry lists 'Last Commit' as 2025.11.06, which is later than the paper's June 2025 revision date; this is likely a typo and should be corrected.
  5. [Appendix C.1] The development-document example is attributed to the tiktoken project, but tiktoken does not appear in the project list in Table 2; please clarify which project this example comes from.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the synthesized tasks are constructive data-generation outputs, and the fine-tuning gain is an in-distribution empirical result rather than a quantity forced by definition.

full rationale

SWE-Flow's derivation chain is constructive: Algorithm 1 executes the project's own passing unit tests and records caller-callee edges; Algorithm 2 orders tasks by the RDG; Section 3.5 builds each partial codebase by deleting TCFN implementations and DCFN definitions; Section 3.6 defines the ground-truth patch as the diff between original and skeletonized code. Verifiability is therefore guaranteed for the reference patch by construction, but this is the intended data-generation mechanism, not a circular inference — the LLM is not given the patch and must reimplement the deleted functions. The fine-tuning evaluation on SWE-Flow-Bench (Lite) uses 12 projects disjoint from the 62 training projects, so the pass-rate improvement measures transfer to held-out tasks from the same generator; it is an empirical, in-distribution measurement, not a fitted parameter renamed as a prediction. The use of Qwen2.5-Coder-32B-Instruct both for generating requirement documents/docstrings and as the base for fine-tuning is a mild same-team self-citation, but the model is open-source and externally available, and the cited technical report is not invoked to forbid alternatives or to supply the central result. Section F.1 candidly acknowledges that the RDG may be incomplete for asyncio and multi-process code; this is a correctness risk about the load-bearing completeness assumption, not a circularity, because the paper does not redefine dependency to make its conclusion true by definition.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No numeric constants are fitted to data in the paper. The design choices (difficulty thresholds at 2 and 3 functions per step, the 50-step Lite split, learning rate 7e-6, max sequence length 32k) are hand-selected but do not enter the central claim as fitted parameters. The core assumptions are about the completeness of the runtime dependency graph and the faithfulness of tests and generated documents as specifications.

assumptions (4)
  • domain assumption Unit tests that pass in the complete codebase are sufficient and faithful specifications of the required functionality for each development step.
    Invoked in Section 3.4 (development document from tests) and Section 3.7 (only passing tests are kept). If tests are incomplete or misleading, the synthesized tasks are ill-posed.
  • domain assumption The runtime dependency graph constructed from the complete, passing test suite fully captures the function interactions required for incremental development.
    Invoked in Sections 3.2 and 3.3; schedule generation and skeletonization rely on complete dependency information. The paper limits this in Section F.1 for async and multiprocessing code.
  • domain assumption Skeletonization that removes function bodies and dependent function definitions leaves a syntactically valid partial codebase that accurately simulates an intermediate development state.
    Invoked in Section 3.5. If removal introduces syntax or import errors unrelated to the task, the benchmark would measure failure to repair broken scaffolding.
  • domain assumption LLM-generated requirement documents from unit tests are faithful, complete specifications of the intended behavior.
    Invoked in Section 3.4. The paper manually reviewed a small sample, but the full dataset is generated automatically, so the assumption carries risk.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Manner." pith.science (2026). https://pith.science/paper/U75YBUGW

@misc{pith2026250609003,
  author       = {Pith},
  title        = {Pith review of: SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Manner},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U75YBUGW}},
  note         = {Machine review of arXiv:2506.09003}
}
read the original abstract

We introduce **SWE-Flow**, a novel data synthesis framework grounded in Test-Driven Development (TDD). Unlike existing software engineering data that rely on human-submitted issues, **SWE-Flow** automatically infers incremental development steps directly from unit tests, which inherently encapsulate high-level requirements. The core of **SWE-Flow** is the construction of a Runtime Dependency Graph (RDG), which precisely captures function interactions, enabling the generation of a structured, step-by-step *development schedule*. At each step, **SWE-Flow** produces a partial codebase, the corresponding unit tests, and the necessary code modifications, resulting in fully verifiable TDD tasks. With this approach, we generated 16,061 training instances and 2,020 test instances from real-world GitHub projects, creating the **SWE-Flow-Eval** benchmark. Our experiments show that fine-tuning open model on this dataset significantly improves performance in TDD-based coding. To facilitate further research, we release all code, datasets, models, and Docker images at [Github](https://github.com/Hambaobao/SWE-Flow).

Figures

Figures reproduced from arXiv: 2506.09003 by the authors.

Figure 1
Figure 1. The framework of SWE-Flow. Step 1: Given a codebase and its corresponding development environment, sweflow executes unit tests, constructs the project’s Runtime Dependency Graph (RDG), and generates a development schedule. Step 2: Based on the development schedule, sweflow removes the implementation of core functions covered by the current step’s test functions, forming an incomplete codebase for development. Additi… view at source ↗
Figure 2
Figure 2. Examples of the Replace Format and Patch Format solution. The left side presents an example of a Replace Format solution, which follows the GitHub merge conflict format. The right side shows an example of a Patch Format solution, which can be directly generated using diff tools. naturally satisfies the topological dependency order. The development schedule P is formally defined as: P = {(FTTFN(i), FTCFN(i), FDCFN(i)… view at source ↗
Figure 3
Figure 3. The evaluation framework of SWE-Flow-Bench. Upper: A prompt containing the current codebase information, development document, and output format is sent to the LLM. The LLM generates a response based on the prompt’s requirements. A post-processing tool then extracts the solution from the LLM’s response, applies it to the codebase, and executes the corresponding unit tests to verify correctness. Lower: Given an incom… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: The overview of evaluation results of large language models on SWE-Flow-Bench (Lite). The x-axis represents the number of development tasks for which the solutions generated by LLMs successfully pass unit tests. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Efficiency values of various LLMs integrated with Open￾Hands. Each axis represents a specific software engineering task, with values indicating the corresponding efficiency scores of the agent for that task. 6. Discussion and Future Work 1. Synthesizing More Challengin…
Figure 6
Figure 6. Figure 6: Content of the target test function and the requirement document generated by Qwen2.5-Coder-32B-Instruct based on it. C.2. Large Language Model Genearted Doc-string For doc-string generation, we adopt the same strategy as used for development document generation. Speci…
Figure 7
Figure 7. Figure 7: Content of the target core function and the doc-string generated by Qwen2.5-Coder-32B-Instruct based on it. --- src/datasets/features/features.py +++ src/datasets/features/features.py @@ -954,7 +954,21 @@ bool: `True` if the object and its first elements (if it is a se…
Figure 8
Figure 8. Figure 8: An example of a reference patch in synthetic data from sweflow. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: The distribution of total tokens In Section 3.7, we described the process of SWE-Flow-synthesized training data generation. Here, we utilize the synthesized dataset to fine-tune the Qwen2.5-Coder-32B-Instruct (Hui et al., 2024) model. The training dataset consists of t…
Figure 10
Figure 10. Figure 10: An Runtime Dependency Graph (RDG) instance form datasets project. src/datasets/formatt ing/formatting.py↦37 9↦format src/datasets/formatt ing/formatting.py↦45 5↦format_column src/datasets/formatt ing/formatting.py↦22 7↦decode_column src/datasets/formatt ing/formatting…
Figure 11
Figure 11. Figure 11: An Runtime Dependency Graph (RDG) instance form datasets project. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: An Runtime Dependency Graph (RDG) instance form datasets project. src/datasets/table.p y↦1343↦_concat_block s_horizontally_and_v ertically src/datasets/table.p y↦1326↦_concat_block s src/datasets/table.p y↦1378↦from_blocks src/datasets/table.p y↦1369↦_consolidate_ blo…
Figure 13
Figure 13. Figure 13: An Runtime Dependency Graph (RDG) instance form datasets project. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories?

    cs.SE 2025-07 conditional novelty 6.0 of 10

    A repository-level benchmark shows LLMs and agents currently achieve far smaller, statistically verified speedups than expert human patches on 140 real-world Python performance tasks.

Reference graph

Works this paper leans on

59 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    Claude 3 family, 2024

    Anthropic. Claude 3 family, 2024. URL https://www.anthropic.com/news/claude-3-family. Accessed: 2025-01-30

    work page 2024

  2. [2]

    Program synthesis with large language models

    Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  3. [3]

    Efficient training of language models to fill in the middle

    Bavarian, M., Jun, H., Tezak, N., Schulman, J., McLeavey, C., Tworek, J., and Chen, M. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207.14255, 2022

    arXiv 2022

  4. [4]

    Test Driven Development: By Example

    Beck, K. Test Driven Development: By Example. Addison-Wesley Longman Publishing Co., Inc., 2002

    work page 2002

  5. [5]

    J., Feldman, M

    Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., Guha, A., Greenberg, M., and Jangda, A. Multipl-e: A scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering, 49 0 (7): 0 3675--3691, 2023. doi:10.1109/TSE.2023.3267446

    arXiv 2023

  6. [6]

    Mceval: Massively multilingual code evaluation

    Chai, L., Liu, S., Yang, J., Yin, Y., Jin, K., Liu, J., Sun, T., Zhang, G., Ren, C., Guo, H., et al. Mceval: Massively multilingual code evaluation. arXiv preprint arXiv:2406.07436, 2024

    arXiv 2024

  7. [7]

    Code alpaca: An instruction-following llama model for code generation

    Chaudhary, S. Code alpaca: An instruction-following llama model for code generation. https://github.com/sahil280114/codealpaca, 2023

    work page 2023

  8. [8]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

Show all 59 references
  1. [9]

    Fullstack bench: Evaluating llms as full stack coders

    Cheng, Y., Chen, J., Chen, J., Chen, L., Chen, L., Chen, W., Chen, Z., Geng, S., Li, A., Li, B., et al. Fullstack bench: Evaluating llms as full stack coders. arXiv e-prints, pp.\ arXiv--2412, 2024

    2024

  2. [10]

    Gu, A., Rozi \`e re, B., Leather, H., Solar-Lezama, A., Synnaeve, G., and Wang, S. I. Cruxeval: A benchmark for code reasoning, understanding and execution. 2024

    2024

  3. [11]

    Deepseek-coder: When the large language model meets programming--the rise of code intelligence

    Guo, D., Zhu, Q., Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y., et al. Deepseek-coder: When the large language model meets programming--the rise of code intelligence. arXiv preprint arXiv:2401.14196, 2024. URL https://arxiv.org/abs/2401.14196

    2024 arXiv

  4. [12]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  5. [13]

    Hong, S., Zheng, X., Chen, J., Cheng, Y., Wang, J., Zhang, C., Wang, Z., Yau, S. K. S., Lin, Z., Zhou, L., et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352, 2023

    2023 arXiv

  6. [14]

    Cosqa: 20,000+ web queries for code search and question answering

    Huang, J., Tang, D., Shou, L., Gong, M., Xu, K., Jiang, D., Zhou, M., and Duan, N. Cosqa: 20,000+ web queries for code search and question answering. 2021

    2021

  7. [15]

    K., Hao, J., Song, L., Xu, Y., Yang, J., Liu, J., Zhang, C., Chai, L., et al

    Huang, S., Cheng, T., Liu, J. K., Hao, J., Song, L., Xu, Y., Yang, J., Liu, J., Zhang, C., Chai, L., et al. Opencoder: The open cookbook for top-tier code large language models. arXiv preprint arXiv:2411.04905, 2024

    2024 arXiv

  8. [16]

    Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Dang, K., et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  9. [17]

    Huq, F., Hasan, M., Haque, M. M. A., Mahbub, S., Iqbal, A., and Ahmed, T. Review4repair: Code review aided automatic program repairing. 143: 0 106765, 2022

    2022

  10. [18]

    P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al

    Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  11. [19]

    Codesearchnet challenge: Evaluating the state of semantic code search

    Husain, H., Wu, H., Gazit, T., Allamanis, M., and Brockschmidt, M. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436, abs/1909.09436, 2019. URL http://arxiv.org/abs/1909.09436

    1909 arXiv

  12. [20]

    Openai o1 system card

    Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  13. [21]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  14. [22]

    E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K

    Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    2023 arXiv

  15. [23]

    Ds-1000: A natural and reliable benchmark for data science code generation

    Lai, Y., Li, C., Wang, Y., Zhang, T., Zhong, R., Zettlemoyer, L., Yih, W.-T., Fried, D., Wang, S., and Yu, T. Ds-1000: A natural and reliable benchmark for data science code generation. ArXiv, abs/2211.11501, 2022

    2022 arXiv

  16. [24]

    B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., Liu, Q., Zheltonozhskii, E., Zhuo, T

    Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., Liu, Q., Zheltonozhskii, E., Zhuo, T. Y., Wang, T., Dehaene, O., Davaadorj, M., Lamy - Poirier, J., Monteiro, J., Shliazhko, O., Gontier, N., Meade, N., Zebaze, A., Y...

    2023 doi arXiv

  17. [25]

    Autokaggle: A multi-agent framework for autonomous data science competitions

    Li, Z., Zang, Q., Ma, D., Guo, J., Zheng, T., Niu, X., Yue, X., Wang, Y., Yang, J., Liu, J., et al. Autokaggle: A multi-agent framework for autonomous data science competitions. arXiv preprint arXiv:2410.20424, 2024

    2024 arXiv

  18. [26]

    Deepseek-v3 technical report

    Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024 a

    2024 arXiv

  19. [27]

    S., Wang, Y., and Zhang, L

    Liu, J., Xia, C. S., Wang, Y., and Zhang, L. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. arXiv preprint arXiv:2305.01210, abs/2305.01210, 2023. URL https://arxiv.org/abs/2305.01210

    2023 arXiv

  20. [28]

    M2rc-eval: Massively multilingual repository-level code completion evaluation

    Liu, J., Deng, K., Liu, C., Yang, J., Liu, S., Zhu, H., Zhao, P., Chai, L., Wu, Y., Jin, K., et al. M2rc-eval: Massively multilingual repository-level code completion evaluation. arXiv preprint arXiv:2410.21157, 2024 b

    2024 arXiv

  21. [30]

    Mdeval: Massively multilingual code debugging

    Liu, S., Chai, L., Yang, J., Shi, J., Zhu, H., Wang, L., Jin, K., Zhang, W., Zhu, H., Guo, S., et al. Mdeval: Massively multilingual code debugging. arXiv preprint arXiv:2411.02310, 2024 d

    2024 arXiv

  22. [31]

    B., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., et al

    Lozhkov, A., Li, R., Allal, L. B., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173, 2024

    2024 arXiv

  23. [32]

    K., Fu, S., and LIU, S

    Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., Li, G., Zhou, L., Shou, L., Zhou, L., Tufano, M., GONG, M., Zhou, M., Duan, N., Sundaresan, N., Deng, S. K., Fu, S., and LIU, S. Code XGLUE : A machine learning benc...

    2021

  24. [33]

    Wizardcoder: Empowering code large language models with evol-instruct

    Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., and Jiang, D. Wizardcoder: Empowering code large language models with evol-instruct. CoRR, abs/2306.08568, 2023. doi:10.48550/ARXIV.2306.08568. URL https://doi.org/10.48550/arXiv.2306.08568

    2023 doi arXiv

  25. [34]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022 a

    2022

  26. [35]

    L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models t...

    2022

  27. [36]

    Training software engineering agents and verifiers with swe-gym

    Pan, J., Wang, X., Neubig, G., Jaitly, N., Ji, H., Suhr, A., and Zhang, Y. Training software engineering agents and verifiers with swe-gym. arXiv preprint arXiv:2412.21139, 2024

    2024 arXiv

  28. [37]

    E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al

    Rozi \`e re, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al. Code Llama : Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. URL https://arxiv.org/abs/2308.12950

    2023 arXiv

  29. [38]

    Toolformer: Language models can teach themselves to use tools

    Schick, T., Dwivedi-Yu, J., Dess \` , R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36: 0 68539--68551, 2023

    2023

  30. [39]

    Megatron-lm: Training multi-billion parameter language models using model parallelism

    Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  31. [40]

    Tablegpt2: A large multimodal model with tabular data integration

    Su, A., Wang, A., Ye, C., Zhou, C., Zhang, G., Zhu, G., Wang, H., Xu, H., Chen, H., Li, H., et al. Tablegpt2: A large multimodal model with tabular data integration. arXiv preprint arXiv:2411.02059, 2024

    2024 arXiv

  32. [41]

    and Nadiri, A

    Talebirad, Y. and Nadiri, A. Multi-agent collaboration: Harnessing the power of intelligent llm agents. arXiv preprint arXiv:2306.03314, 2023

    2023 arXiv

  33. [42]

    Debugbench: Evaluating debugging capability of large language models

    Tian, R., Ye, Y., Qin, Y., Cong, X., Lin, Y., Liu, Z., and Sun, M. Debugbench: Evaluating debugging capability of large language models. 2024

    2024

  34. [44]

    F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., et al

    Wang, X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., et al. Opendevin: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741, 2024 b

    2024 arXiv

  35. [45]

    A., Khashabi, D., and Hajishirzi, H

    Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions. In Rogers, A., Boyd - Graber, J. L., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Associa...

    2023 doi

  36. [46]

    Magicoder: Source code is all you need

    Wei, Y., Wang, Z., Liu, J., Ding, Y., and Zhang, L. Magicoder: Source code is all you need. CoRR, abs/2312.02120, 2023. doi:10.48550/ARXIV.2312.02120. URL https://doi.org/10.48550/arXiv.2312.02120

    2023 doi arXiv

  37. [47]

    Tablebench: A comprehensive and complex benchmark for table question answering

    Wu, X., Yang, J., Chai, L., Zhang, G., Liu, J., Du, X., Liang, D., Shu, D., Cheng, X., Sun, T., et al. Tablebench: A comprehensive and complex benchmark for table question answering. arXiv preprint arXiv:2408.09174, 2024

    2024 arXiv

  38. [48]

    Codetransocean: A comprehensive multilingual benchmark for code translation

    Yan, W., Tian, Y., Li, Y., Chen, Q., and Wang, W. Codetransocean: A comprehensive multilingual benchmark for code translation. 2023

    2023

  39. [49]

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024 a

    2024 arXiv

  40. [50]

    E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., and Press, O

    Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., and Press, O. Swe-agent: Agent-computer interfaces enable automated software engineering. arXiv preprint arXiv:2405.15793, 2024 b

    2024 arXiv

  41. [51]

    Evaluating and aligning codellms on human preference

    Yang, J., Yang, J., Jin, K., Miao, Y., Zhang, L., Yang, L., Cui, Z., Zhang, Y., Hui, B., and Lin, J. Evaluating and aligning codellms on human preference. arXiv preprint arXiv:2412.05210, 2024 c

    2024 arXiv

  42. [52]

    Execrepobench: Multi-level executable code completion evaluation

    Yang, J., Zhang, J., Yang, J., Jin, K., Zhang, L., Peng, Q., Deng, K., Miao, Y., Liu, T., Cui, Z., et al. Execrepobench: Multi-level executable code completion evaluation. arXiv preprint arXiv:2412.11990, 2024 d

    2024 arXiv

  43. [53]

    Codereval: A benchmark of pragmatic code generation with generative pre-trained models

    Yu, H., Shen, B., Ran, D., Zhang, J., Zhang, Q., Ma, Y., Liang, G., Li, Y., Wang, Q., and Xie, T. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, pp.\ ...

    2024

  44. [54]

    Wavecoder: Widespread and versatile enhanced instruction tuning with refined data generation

    Yu, Z., Zhang, X., Shang, N., Huang, Y., Xu, C., Zhao, Y., Hu, W., and Yin, Q. Wavecoder: Widespread and versatile enhanced instruction tuning with refined data generation. CoRR, abs/2312.14187, 2023. doi:10.48550/ARXIV.2312.14187. URL https://doi.org/10.48550/arXiv.2312.14187

    2023 doi arXiv

  45. [55]

    RepoCoder : Repository-level code completion through iterative retrieval and generation

    Zhang, F., Chen, B., Zhang, Y., Liu, J., Zan, D., Mao, Y., Lou, J., and Chen, W. RepoCoder : Repository-level code completion through iterative retrieval and generation. arXiv preprint arXiv:2303.12570, abs/2303.12570, 2023 a . doi:10.48550/arXiv.2303.12570. URL https://doi.or...

    2023 doi arXiv

  46. [56]

    Di-bench: Benchmarking large language models on dependency inference with testable repositories at scale

    Zhang, L., Wang, J., He, S., Zhang, C., Kang, Y., Li, B., Wen, J., Xie, C., Wang, M., Huang, Y., et al. Di-bench: Benchmarking large language models on dependency inference with testable repositories at scale. arXiv preprint arXiv:2501.13699, 2025

    2025 arXiv

  47. [57]

    Llama-adapter: Efficient fine-tuning of language models with zero-init attention

    Zhang, R., Han, J., Zhou, A., Hu, X., Yan, S., Lu, P., Li, H., Gao, P., and Qiao, Y. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. CoRR, abs/2303.16199, 2023 b . doi:10.48550/ARXIV.2303.16199. URL https://doi.org/10.48550/arXiv.2303.16199

    2023 doi arXiv

  48. [58]

    Naturalcodebench: Examining coding performance mismatch on humaneval and natural user queries

    Zhang, S., Zhao, H., Liu, X., Zheng, Q., Qi, Z., Gu, X., Dong, Y., and Tang, J. Naturalcodebench: Examining coding performance mismatch on humaneval and natural user queries. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual ...

    2024

  49. [59]

    T., Cardie, C., Gall \'e , M., and Rush, A

    Zhao, W., Jiang, N., Lee, C., Chiu, J. T., Cardie, C., Gall \'e , M., and Rush, A. M. Commit0: Library generation from scratch. arXiv preprint arXiv:2412.01769, 2024

    2024 arXiv

  50. [60]

    Y., Vu, M

    Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877, 2024

    2024 arXiv

  51. [61]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.