{"id":"5e118294-1277-4a53-895f-6b2d6e5c48df","arxiv_id":"2506.09003","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SWE-Flow synthesizes incremental, test-driven development tasks from real GitHub projects and shows that fine-tuning Qwen2.5-Coder-32B-Instruct on them improves performance on the resulting SWE-Flow-Bench benchmark.","lead":"SWE-Flow builds software engineering training data by running unit tests, tracing function call dependencies, and turning codebases into step-by-step test-driven development tasks. The authors release a benchmark, SWE-Flow-Bench, and a fine-tuned code model that outperforms its base model on those tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RDG completeness is the load-bearing assumption; the schedule and verifiability claims both depend on it, and the current validation does not measure missed-edge rates.","rationale":"The reader's weakest_assumption is exactly the RDG completeness assumption, and I agree that it is load-bearing. My check sharpens it into an operational, falsifiable audit: applying the reference patch to the skeletonized codebase and running the step's tests directly measures whether the schedule and skeletonization preserve verifiability. The reader's other concerns (self-referential benchmark, patch-to-replace post-processing) are real but secondary; the deficiency I identify is more basic, because if a non-trivial fraction of tasks are not actually solvable from the skeleton, the benchmark numbers and the fine-tuning benefit are both called into question. I also note the paper provides no failure-mode statistics: Section F.1 lists limitation classes but no quantification, and Section 3.7's 'only keep unit tests that pass' refers to the original project, not the synthetic per-step state. The independent evidence the paper does provide — released code, data, models, Docker images — makes the proposed audit feasible and is why the correct verdict is CONDITIONAL rather than REJECT: the central framework is sound in principle and externally checkable, but the central claim as stated is currently under-validated. I would require the audit as a condition of acceptance, along with the reader's requested naming consistency fixes and external-benchmark generalization check.","tokens_in":24944,"tokens_out":2363,"duration_ms":19251,"concrete_test":"Reproduce the verifiability audit that the paper omits: for every task in SWE-Flow-Bench (Lite) and a random sample of training tasks, apply the ground-truth patch to the skeletonized codebase, install the container, and run the step's associated unit tests. Record (a) the fraction of tasks that pass their own tests after applying the ground truth (expected 100% if the schedule is correct), and (b) for failures, whether the failure is a NameError/ImportError for a function not declared in the step's newFuncNodes (a missed RDG edge) or a semantics mismatch. If (a) is substantially below 100%, the 'fully verifiable' and 'incremental build order' claims fail and the evaluation numbers are best-case; if (a) is 100% on a representative sample, the RDG-completeness concern is empirically settled for the supported synchronous subset.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim — that every SWE-Flow task is fully verifiable and that the development schedule is incrementally buildable — rests on the completeness of the Runtime Dependency Graph. Algorithm 1 records only direct, synchronous, in-process caller-callee edges observed while executing the project's own already-passing test suite. Section F.1 concedes that asyncio, multi-process, and IPC patterns are not tracked; C extensions, reflection, eval/exec, and dynamic dispatch (e.g., __getattr__ delegation, mock/parametrize) are also invisible to a Python-level sys.settrace hook. The schedule in Algorithm 2 orders steps by the size of the CFN set per test, assuming that whenever a later-step TTFN is executed it can only reach functions already developed plus the step's own new CFNs. If the RDG misses an edge (e.g., a later test imports a helper that is neither in an earlier developed set nor declared as a new node for that step), the skeletonized codebase at that step will be missing definitions, and the task is either unsolvable or solvable only by guessing the hidden dependency. The paper reports pass rates for the benchmark and for the fine-tuned model, and Table 3 reports dependency depth and counts, but it never reports how often a task is actually re-executable from the skeletonized codebase, nor how many tasks were discarded because a later test failed due to missing undeclared dependencies. 'We only keep the unit tests that pass' (Section 3.7) filters on the original full project, not on the skeletonized per-step state, so it does not validate the schedule. Without a check that every synthesized task's tests pass on the skeleton-plus-ground-truth-patch, the 'fully verifiable' claim is unestablished, and a subset of tasks could be degenerate. The fine-tuning gain could then partly reflect learning to repair broken skeletons rather than genuine TDD development.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SWE-Flow, a framework that synthesizes test-driven development (TDD) training and evaluation data from real-world GitHub repositories. The framework executes a project's unit tests under a Python tracing hook to build a Runtime Dependency Graph (RDG), derives a development schedule that orders function implementations by dependency size, and then skeletonizes the original codebase for each step by removing the implementations of target and dependent core functions. For each step it generates a requirement document, a partial codebase, and ground-truth patches in both replace and patch formats. The authors release 16,061 training instances and 2,020 test instances, introduce SWE-Flow-Bench, and fine-tune Qwen2.5-Coder-32B-Instruct on the synthesized data. They report that the fine-tuned model, SF-Coder-32B-Instruct, outperforms the base model and most closed models on SWE-Flow-Bench (Lite) in both replace and patch formats, and they claim that the synthesized data are fully verifiable because correctness is checked by executing unit tests.","tokens_in":25223,"tokens_out":6549,"duration_ms":67646,"significance":"If the central claims hold, SWE-Flow addresses a real bottleneck in software engineering data: the dependence on human-submitted issue reports. The idea of deriving incremental development steps from unit tests and using execution-based verification is timely and potentially scalable, and the public release of code, datasets, models, and Docker images is a concrete asset to the community. The paper also provides explicit algorithms (Algorithm 1 and Algorithm 2), which makes the method reproducible in principle. However, the current evidence does not fully support the 'fully verifiable' claim, the patch-format evaluation is weakened by a post-processing conversion, and the fine-tuning gains are measured only on a benchmark generated by the same pipeline as the training data. These issues are fixable with additional validation and clearer reporting, but they are load-bearing for the paper's main conclusions.","major_comments":[{"comment":"The claim that every SWE-Flow task is 'fully verifiable' is not directly validated. The statement 'We only keep the unit tests that pass' (Section 3.7) refers to tests executed on the original full codebase, not to tests executed on each skeletonized codebase after applying the ground-truth patch. Because skeletonization removes entire function definitions (DCFNs) and replaces TCFN bodies with LLM-generated docstrings, the skeletonized codebase may fail to import or may otherwise not be re-executable even after the reference patch is applied. Please report, for every benchmark task, the result of applying the reference patch to the skeletonized codebase and running the step's unit tests, along with the number of tasks discarded due to import errors or undeclared dependencies. Without this measurement, the 'fully verifiable' property is an assumption rather than a demonstrated fact.","section":"Section 3.7 and Algorithm 1"},{"comment":"The 'Patch Format' results do not measure the quality of generated patches as patches. Appendix E.1 states that generated patches are 'almost entirely incompatible with system tools such as Linux's patch utility' and that the evaluation applies a post-processing step that converts generated patches into replace format before modifying the codebase. This means the patch-format pass rates in Table 5 and Figure 4 reflect the success of an internal repair/conversion process, not the model's ability to produce an applicable unified diff. Either evaluate patch-format outputs with an actual patch application tool that performs context matching, or clearly relabel the metric and report how often the conversion changes the outcome.","section":"Section 4.1 and Appendix E.1"},{"comment":"The empirical validation of fine-tuning is circular in an important sense: SWE-Flow-Bench is generated by the same SWE-Flow pipeline that produced the training data. The test projects are disjoint from training projects, which mitigates direct memorization, but the task distribution, prompt format, skeletonization style, and requirement-document generation are identical. The reported improvement therefore demonstrates that the model learns the SWE-Flow task distribution, not that it improves general TDD-based coding ability. A comparison on an external benchmark such as SWE-Bench, Commit0, or a human-validated subset of real development tasks would substantially strengthen the claim that the synthesized data improve software engineering capability beyond the synthetic distribution.","section":"Section 5.2 and Section 4.1"},{"comment":"Skeletonization of Dependent Core Function Nodes removes both the function definition and its contents. A removed function that is referenced at module import time by code outside the current step (for example, as a decorator, a default argument, or a module-level assignment) will cause the skeletonized codebase to fail import, and no patch that only re-implements the current step's new functions can repair that. This is a concrete failure mode that is not captured by running tests on the original codebase and is not discussed in Section F.1, which lists only asyncio and multi-process limitations. The proposed per-task import-and-test validation in Section 3.7 would directly expose this issue.","section":"Section 3.5"},{"comment":"The paper acknowledges that the RDG is incomplete for asynchronous and multi-process programs, but it never quantifies how often such incompleteness affects the synthesized tasks. Table 3 reports dependency depth and function counts, none of which measure missed-edge rates. Please provide a quantitative analysis of RDG completeness, for example by comparing the runtime graph against a static call graph (with appropriate caveats) or by reporting the number of steps whose tests fail on the skeletonized codebase even after the reference patch is applied. This is needed to assess the reliability of the development schedule and the 'fully verifiable' claim.","section":"Section F.1 and Table 3"}],"minor_comments":[{"comment":"The model name is inconsistent: the text refers to SF-Coder-32B-Instruct, while Figure 4 uses both UF-Coder-32B-Instruct and UF-Coder-v4.0-iters-32. Please unify the naming across the paper.","section":"Section 5.2 and Figure 4"},{"comment":"The benchmark is called SWE-Flow-Eval in the abstract but SWE-Flow-Bench throughout the body. Please use one name consistently.","section":"Abstract and Section 3.7"},{"comment":"The pylint Full Context Tokens value is listed as '23,76', which appears to be a typo for '2,376'.","section":"Table 3"},{"comment":"The marshmallow entry lists 'Last Commit' as 2025.11.06, which is later than the paper's June 2025 revision date; this is likely a typo and should be corrected.","section":"Table 2"},{"comment":"The development-document example is attributed to the tiktoken project, but tiktoken does not appear in the project list in Table 2; please clarify which project this example comes from.","section":"Appendix C.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is sound and the release is a strong contribution, but the 'fully verifiable' claim and the patch-format evaluation need to be hardened before publication. The authors should be asked to supply per-task verification results on the skeletonized codebases and either re-run the patch-format evaluation with a real patch tool or clearly relabel the metric. If those additions confirm the pipeline's reliability, the paper would be a solid contribution to the machine learning for code community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper has a genuinely new way to synthesize incremental TDD training data from unit tests, and it ships the code, data, and a fine-tuned model. The central empirical claim—fine-tuning on this data improves TDD coding—is probably true in the limited sense the paper measures, but the evidence is mostly self-referential, and the \"fully verifiable\" guarantee is not actually checked.\n\nWhat's new: the RDG-based schedule generation plus skeletonization into per-step tasks is a real departure from SWE-Bench and Commit0. Using the runtime call stack to order development steps, then stripping dependent functions and generating a docstring for target functions, is a clever and reproducible recipe. The scale is substantial: 16k training instances, 2k test instances from 74 projects, and a fine-tuned Qwen2.5-Coder-32B that beats the base model and most closed models on their own Lite set. The release of artifacts is a real asset.\n\nThe soft spots, in rough order of importance. First, the whole pipeline rests on the Runtime Dependency Graph being complete enough that the generated schedule is actually buildable. Direct, synchronous, in-process calls captured by sys.settrace miss asyncio, multiprocessing, C extensions, reflection, dynamic dispatch. The authors concede asyncio and multiprocessing in Section F.1. If an edge is missing, a later step can require a function that is neither in earlier developed sets nor declared as new, and the skeleton code is broken. Critically, they never report whether the ground-truth patch applied to the skeleton actually passes the step's tests. The \"we only keep unit tests that pass\" filter is on the full original project, not on the skeletonized per-step state. That means \"fully verifiable\" is asserted, not demonstrated.\n\nSecond, the evaluation is on a benchmark produced by the same pipeline, so the fine-tuned model is measured on the exact task distribution it was trained on. Project-level disjointness helps avoid direct memorization, but it doesn't show generalization to other task distributions. An external benchmark or a human-authored TDD set would strengthen it a lot. Third, patch-format results are actually generated patches converted into replace format in post-processing because system patch tools rejected them—so the patch numbers aren't measuring what a reader might assume. Fourth, naming inconsistencies (SWE-Flow-Eval vs SWE-Flow-Bench, SF-Coder vs UF-Coder) are minor but sloppy.\n\nOverall, I think the core idea is sound and the limitations are honestly acknowledged. The main missing piece is a validation step showing every synthesized task is solvable, plus at least one external evaluation. This deserves a serious referee—the direction is important for synthetic code data—but it should come back with that extra evidence.","headline":"A genuinely new pipeline for synthesizing incremental TDD data from unit tests, with honest limitations, but the 'fully verifiable' guarantee is asserted rather than demonstrated and the benchmark is self-referential.","tokens_in":25829,"tokens_out":2249,"would_cite":true,"duration_ms":21752,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SWE-Flow synthesizes fully verifiable test-driven software engineering data from unit tests alone, and fine-tuning a 32B open code model on that data lifts it past most closed models on incremental coding tasks.","keywords":["test-driven development","software engineering data synthesis","runtime dependency graph","unit tests","code model fine-tuning","incremental development","verifiable training data","code generation benchmark"],"falsifier":"Run SWE-Flow on a project that uses asyncio, multiprocessing, or callback-heavy patterns; for every scheduled step, apply the step's ground-truth patch to the skeletonized codebase and run that step's unit tests. If any step fails its own tests or fails to import because a needed function is missing, the RDG missed an edge and the verifiability claim is broken — the pattern the authors themselves flag in their limitations section.","tokens_in":24636,"feed_emoji":"🧪","tokens_out":9221,"duration_ms":74681,"temperature":0.7,"pith_summary":"SWE-Flow claims that high-quality software engineering training data can be synthesized automatically from unit tests alone, without human-submitted issue reports. It executes a project's passing tests with a runtime hook, records the function-call relationships into a Runtime Dependency Graph, and derives a step-by-step development schedule where each step adds exactly the functions that a set of tests exercises. For every step it produces a skeletonized partial codebase, a requirement document generated from the test contents, and the reference diff back to the full code, making every task executable and verifiable by its unit tests. From 74 real GitHub projects it generates 16,061 training instances and a 2,020-task benchmark, and fine-tuning an open 32B code model on those instances improves its pass rate on the benchmark over the base model and over most closed models in both replace and patch formats.","feed_headline":"SWE-Flow data lifts 32B model past most closed coding models","feed_subtitle":"Unit tests alone generate 16,061 verifiable coding tasks; fine-tuning on them beats base and most closed models.","key_machinery":"The Runtime Dependency Graph (RDG), built by SWE-Flow-Trace: a directed graph whose nodes are project functions invoked during unit test execution and whose edges are direct caller-callee pairs recorded from the function call stack, with calls outside the project filtered out. SWE-Flow-Schedule consumes the RDG, groups target test functions by the core functions they cover, sorts the groups by coverage size, and merges already-covered functions into earlier groups, producing a development schedule that respects dependency order. Skeletonization then turns each schedule step into a task: target core functions have their bodies removed and replaced by an LLM-generated docstring, while dependent core functions are deleted entirely, and the diff between the skeletonized and original codebases becomes the ground-truth solution.","core_discovery":"The central claim is that the incremental development history of a codebase can be reconstructed from its unit tests: each test is treated as a requirement, the functions it invokes form a Runtime Dependency Graph (RDG), and the topological order of that graph yields a natural development schedule. SWE-Flow-Trace captures the graph by monitoring the live function call stack while all passing tests run, filtering out calls outside the project; SWE-Flow-Schedule then merges tests that cover the same core functions and sorts them by the number of core functions involved, so each scheduled step rests only on functions developed in earlier steps. The paper maintains that every synthesized task is fully verifiable because correctness is decided by executing the step's unit tests in a containerized environment, and that this verifiability, together with automatic inference from real projects, removes the reliance on curated human issue reports. The empirical claim is that fine-tuning Qwen2.5-Coder-32B-Instruct on this data produces a model whose pass rates on SWE-Flow-Bench (Lite) improve on the base model and exceed those of most evaluated closed models in both the replace and patch solution formats.","pith_inferences":["Editorial inference: part of the fine-tuning gain on SWE-Flow-Bench (Lite) could come from memorization, since the tasks are derived from real open-source code that the base model may already have seen; a contamination-controlled split using recently written or private projects would isolate the benefit of the data format itself.","Editorial inference: the runtime-tracing machinery is not Python-specific in principle, so porting the hook approach to other runtimes with call-stack inspection (e.g., a JavaScript engine inspector) would test whether the RDG-schedule recipe transfers to other ecosystems.","Editorial inference: the quality of a generated requirement document is bounded by the expressiveness of the unit test, so tests relying on mocking, fixtures, or external services may yield specifications that are under- or over-constrained relative to the intended function.","Editorial inference: a direct probe of the framework's core claim is to delete one dependency edge from the RDG before scheduling and check whether any step becomes unsolvable; the fraction of failures would quantify how much verifiability depends on graph completeness."],"forward_implications":["Fine-tuning open code models on SWE-Flow data improves their performance on unseen development tasks drawn from different projects, in both the replace and patch output formats.","Any GitHub project with an installable environment and a passing unit test suite becomes a source of verifiable training data, removing the dependence on human-submitted issues and manual filtering.","Because every task is verified by executing unit tests, the same pipeline can supply verifiable reward signals for reinforcement learning and large corpora for pre-training code models.","Merging consecutive schedule steps yields harder tasks, up to building an entire project from scratch, giving a configurable difficulty control for training and evaluation.","SWE-Flow-Bench grades incremental, test-guided development steps, complementing SWE-Bench's issue-patch evaluation and Commit0's one-shot full-library generation."],"supporting_citations":[{"why":"Supplies the Test-Driven Development methodology that motivates deriving requirements and incremental steps from unit tests.","marker":"(Beck, 2002)"},{"why":"SWE-Bench is the prior benchmark the paper positions against and whose evaluation style (apply patch, run tests) it adapts.","marker":"(Jimenez et al., 2023)"},{"why":"Qwen2.5-Coder-32B-Instruct is the base model fine-tuned on the data and the model used to generate requirement documents and docstrings.","marker":"(Hui et al., 2024)"},{"why":"Commit0-Bench is the contrast benchmark requiring one-shot full implementation, which SWE-Flow-Bench is designed to improve upon.","marker":"(Zhao et al., 2024)"},{"why":"Supplies the pass-rate evaluation metric used to measure task success.","marker":"(Chen et al., 2021)"},{"why":"GPT-4o is one of the closed models compared in the benchmark and one of the candidates assessed for document generation.","marker":"(Hurst et al., 2024)"},{"why":"Claude 3.5 Sonnet is the strongest closed model compared and the strongest agent backbone in the OpenHands evaluation.","marker":"(Anthropic, 2024)"},{"why":"DeepSeek-V3 is a compared closed model and was assessed for development-document quality.","marker":"(Liu et al., 2024a)"},{"why":"OpenHands is the agent framework used to evaluate agent-based performance on SWE-Flow-Bench.","marker":"(Wang et al., 2024a)"}],"fun_headline_variants":["Unit tests spawn 16K coding tasks, fine-tuned model beats closed ones","From tests to tasks: SWE-Flow auto-builds coding benchmarks","No human issues needed: SWE-Flow generates verifiable coding data","SWE-Flow turns unit tests into training data, lifts model past closed rivals","Automated TDD data synthesis yields model that outdoes closed systems"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Runtime Dependency Graph captured from the complete, passing test suite reflects every function interaction needed to rebuild the project incrementally, so that each scheduled step's partial codebase is buildable and its tests are passable.","fun_headline_variants_meta":{"raw":{"variants":["Unit tests spawn 16K coding tasks, fine-tuned model beats closed ones","From tests to tasks: SWE-Flow auto-builds coding benchmarks","No human issues needed: SWE-Flow generates verifiable coding data","SWE-Flow turns unit tests into training data, lifts model past closed rivals","Automated TDD data synthesis yields model that outdoes closed systems"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000319,"raw_usage":{"total_tokens":1821,"prompt_tokens":987,"completion_tokens":834,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":735}},"tokens_in":603,"tokens_out":834,"duration_ms":8156,"temperature":1.0,"reasoning_tokens":735,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:56:54.893047+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SWE-Flow on a project that uses asyncio, multiprocessing, or callback-heavy patterns; for every scheduled step, apply the step's ground-truth patch to the skeletonized codebase and run that step's unit tests. If any step fails its own tests or fails to import because a needed function is missing, the RDG missed an edge and the verifiability claim is broken — the pattern the authors themselves flag in their limitations section.","supporting_citations":[{"cited_title":"Test Driven Development: By Example","cited_arxiv_id":null,"evidence_quote":"Supplies the Test-Driven Development methodology that motivates deriving requirements and incremental steps from unit tests."},{"cited_title":"Claude 3 family, 2024","cited_arxiv_id":null,"evidence_quote":"Claude 3.5 Sonnet is the strongest closed model compared and the strongest agent backbone in the OpenHands evaluation."}],"review_version":1}