{"id":"87ab7a0f-5853-45a7-a7e4-e6806ff27b03","arxiv_id":"2608.09072","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SWE-RPG is a new benchmark with validated references for requirement clarification and planning, showing that coding agents fail most often at recovering implicit requirements.","lead":"This paper introduces SWE-RPG, a new benchmark of 163 real coding tasks that scores AI coding agents not only on whether their final patch passes tests, but also on how well they clarify hidden requirements and plan before coding. The authors report that agents resolve only 31.5% of tasks on average, with recovering implicit requirements being the largest bottleneck.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The bottleneck claim rests on an LLM judge whose 92% agreement is measured on n=50 unresolved runs, with no per-stage confusion or inter-annotator reliability; if the judge systematically over-attributes failures to the requirement stage, the central diagnostic conclusion would be an artifact.","rationale":"The reader's weakest assumption is exactly the measurement validity of the LLM-generated Clarification and Plan GTs and the LLM-judge stage attributions, and I agree that this is the load-bearing point. The benchmark has real independent support: executable environments are validated with fail-to-pass and pass-to-pass tests, Planning GTs are checked by verifying that code generated from each step is functionally equivalent to the gold subpatch, Clarification GTs are evidence-checked and manually reviewed by two authors, and the artifacts are public. Patch correctness evaluation is therefore not the concern. The concern is narrower but decisive for the paper's central contribution: the claim that requirement recovery is the main bottleneck is causally downstream of a single LLM judge validated on 50 runs against an unspecified 'human consensus,' with no inter-annotator agreement, no per-stage confusion matrix, and no blind condition to rule out anchor effects from the LLM-synthesized GTs. The paper itself flags this in Limitations, listing inter-annotator agreement as future work, which supports treating the diagnostic claim as provisional. The reader's CONDITIONAL verdict is appropriate; my analysis does not move it, but it specifies an experiment that would settle whether the central finding survives. If independent human relabeling reproduces the judge's stage distribution, the bottleneck claim is substantially strengthened. If it does not, the paper's recommendation should be downgraded to a hypothesis pending measurement validation.","tokens_in":12327,"tokens_out":3274,"duration_ms":33367,"concrete_test":"Re-annotate a stratified random sample of 120–150 unresolved runs (balanced across agent, LLM, and preliminary judge label) with two independent human annotators who see only the original issue, repository context, and trajectory excerpt—not the synthesized Clarification/Plan GTs and not the judge's labels. Compute per-stage confusion, Cohen's kappa between annotators, and the requirement-failure share under human labels. Also run the GPT-5.6-Sol judge in a blind ablation (no GT references) on a matched subsample. If the human requirement-failure share falls below 24.5% or no longer ranks first, or if kappa is below 0.6, the bottleneck conclusion should be revised or explicitly reported as judge-dependent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline diagnostic claim—implicit-requirement recovery is the main bottleneck, accounting for 24.5%–46.0% of runs—depends entirely on failure-attribution labels produced by the GPT-5.6-Sol judge in Section 4. The only validation reported is: 'On a stratified sample of 50 unresolved runs, its labels exactly match human consensus in 46 cases (92%).' This is insufficient to establish the claimed bottleneck. First, exact-match accuracy of 92% on n=50 gives a wide confidence interval, and per-stage error rates are unreported; a judge can be 92% accurate overall yet systematically miscount one stage if its errors concentrate there. Second, the judge is given the same kind of LLM-synthesized intermediate GTs (built with GPT-5.4 in Section 3.2) that it is evaluating against, so there is no blind condition or ablation to show that the GTs do not anchor the judge toward 'requirement failure' whenever trajectory evidence is thin. Third, the 'earliest deviating stage' protocol prioritizes any early omission, even one that never became consequential, as a requirement failure, inflating that category relative to planning and code generation. The paper's own Limitations section concedes that the retrospective alignment labels 'depend on LLM-assisted construction and judging' and lists inter-annotator agreement as future work. Because the stage-ranking is the paper's main actionable finding, measurement validity is the load-bearing concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SWE-RPG, a repository-level benchmark of 163 tasks from 31 Python and Java repositories that combines executable patch evaluation with ground-truth references for requirement clarification and implementation planning. The authors evaluate three coding-agent frameworks paired with six LLM backends, reporting an average resolved rate of 31.5%. Using a GPT-5.6-Sol judge to attribute unresolved runs to failure stages, they conclude that implicit requirement recovery is the main bottleneck, accounting for 24.5%–46.0% of agent runs. The benchmark also reports pre-code coverage rates of clarification categories and planning dimensions for the three agents.","tokens_in":12580,"tokens_out":5782,"duration_ms":50503,"significance":"If the measurements are valid, this is a valuable contribution: SWE-RPG is, to my knowledge, the first repository-level benchmark that jointly evaluates requirement clarification, planning, and code generation with validated intermediate references, and the stage-attributed failure analysis goes beyond the pass/fail verdicts of SWE-bench-style benchmarks. The authors provide a reproducible pipeline, executable environments, and public code and data, which are concrete strengths. The paper also offers actionable hypotheses, such as agents being weakest on interface, structure, and data-semantics requirements and on planning constraints/validation. However, the central diagnostic claims rest on LLM-generated ground truths and LLM-judged trajectory labels whose validity is only partially demonstrated; the reported validation samples are small and lack per-stage and per-category breakdowns. The resource itself is likely to be useful regardless, but the headline bottleneck conclusion needs stronger measurement evidence before it can be accepted.","major_comments":[{"comment":"The paper's central diagnostic claim—that implicit requirement recovery is the main bottleneck, accounting for 24.5%–46.0% of runs (Figure 6 and Section 5.2)—rests entirely on labels produced by the GPT-5.6-Sol judge, yet the only reported validation is exact-match agreement of 92% on a stratified sample of 50 unresolved runs. This is insufficient because exact-match accuracy on n=50 has a wide confidence interval, per-stage error rates are not reported, and a judge can be 92% accurate overall while systematically overcounting one stage if its errors concentrate there. In addition, the judge is given the same LLM-synthesized Clarification and Plan GTs (Section 3.2) that it evaluates against, with no blind condition or ablation to rule out anchoring toward 'requirement failure' when trajectory evidence is thin; the 'earliest deviating stage' protocol also counts any early omission as a requirement failure even if it never became consequential. The Limitations section itself concedes that 'retrospective alignment labels depend on LLM-assisted construction and judging' and lists inter-annotator agreement as future work. Please provide per-stage confusion matrices, precision/recall with confidence intervals, a second human annotator, and a sensitivity analysis that re-attributes failures under an alternative protocol such as 'most consequential deviation.'","section":"Section 4, 'Failure Attribution to Stages'"},{"comment":"The Clarification and Plan GTs are synthesized by GPT-5.4 and then 'two authors independently review every resulting reference for evidential support and cross-stage consistency,' but no quantitative inter-annotator agreement is reported, and the paper does not specify how 'evidential support' was operationalized or whether reviewers could detect a synthesized QA pair that is plausible but not actually necessary for the task. Because the diagnostic judge in Section 4 compares trajectories to these GTs, any invented or misweighted implicit requirement directly shifts the stage-attribution distribution. Please report annotator agreement on the GTs, provide evidence-provenance traces showing which repository artifact supports each QA pair and plan step, and, if feasible, run a sensitivity analysis that re-runs diagnosis with a subset of GTs removed or with an independently constructed GT set.","section":"Section 3.2, 'Ground-Truth Construction for Req. Clarification' and 'Ground-Truth Construction for Planning'"},{"comment":"The 96% agreement figure for the judge's binary coverage decisions is based on 'a manually annotated sample of 50 information-point assessments,' but no per-category breakdown is given. Figure 7's finding that agents are weakest on C4–C6 (interface, structure, data semantics) and on planning constraints/validation could be an artifact of judge bias on exactly those categories if the judge is more likely to mark an item uncovered for categories with less textual evidence in the trajectory. Please report per-category agreement, per-category error rates, and inter-annotator reliability for the coverage judgments; otherwise the capability-ranking claims in Section 5.3 are not sufficiently supported.","section":"Section 4, 'Pre-code Stage Evaluation'"}],"minor_comments":[{"comment":"The statement 'We run each agent-LLM configuration twice on every task and report run-level aggregates over the two runs' is ambiguous; please state explicitly whether the reported resolve rate is per run or per task (e.g., a task counts as resolved if either run succeeds), since all reported percentages depend on this choice.","section":"Section 5.1, 'Effectiveness of Agents, LLMs, and Pairings'"},{"comment":"The column header 'Plan GT Statistic' appears to label rows that are task-level statistics (Requirement Words, Codebase LOC), while the plan-specific rows follow below; consider separating task-level and GT-level statistics or adjusting the header to avoid confusion.","section":"Table 3"},{"comment":"The stacked bars contain many dense percentage labels that are hard to read; consider presenting the numerical values in a table as well, and define the 'Others' category in the caption (the legend already mentions timeouts and patch format, but the caption should be self-contained).","section":"Figure 6"},{"comment":"The validation agent is said to check for 'absence of implementation detail leakage,' but the paper does not define what counts as leakage or how the check was enforced; please clarify this criterion.","section":"Section 3.2, 'Synthesis and Validation'"}],"recommendation":"major_revision","confidential_remarks":"The paper's contribution is potentially significant, and the authors have done substantial engineering work in building environments, validating tests, and releasing code and data. My main concern is that the headline diagnostic result—requirement recovery as the dominant bottleneck—rests on a measurement chain (GPT-5.4-synthesized GTs, GPT-5.6-Sol judge, small validation samples) whose validity is not yet demonstrated at the level the claim requires. The requested analyses (per-stage confusion, inter-annotator agreement, sensitivity to the attribution protocol) are feasible within the scope of a revision. I also recommend that the authors be careful in how they position the claim of being 'first' relative to RACE-bench, since the distinction depends on the specific definition of implicit-requirement references and plan-to-code reproducibility; I do not see this as a blocker, but it should be stated crisply."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nSWE-RPG is a real step forward for coding-agent evaluation. It delivers something prior benchmarks don't: validated intermediate ground truths for implicit requirement clarification and implementation planning, plus a test of whether each plan step can actually reproduce its gold subpatch. The construction pipeline is serious — 2,000+ candidates filtered down to 163 with reproducible Docker environments and stable tests, and every resulting GT was manually reviewed by two authors. Compared to RACE-bench and Dialogue SWE-Bench, the distinction is genuinely novel: RACE doesn't provide references for naturally implicit requirements, and Dialogue SWE-Bench creates synthetic omissions rather than mining real ones. The evaluation of 3 agents × 6 LLMs with cost/time data is also solid, and the differences in stage profiles between similarly-performing pairings (e.g., Codex–DSV4 vs. Claude–DSV4) show that outcome-only evaluation is missing signal.\n\nThe soft spot is exactly where the reader and the stress test put it. The central diagnostic finding — that implicit requirement recovery is the dominant bottleneck, at 24.5–46% of runs — rests on attribution labels from a single LLM judge (GPT-5.6-Sol). The only validation reported is 92% exact match on 50 unresolved runs. That's not enough. There's no per-stage confusion matrix, no inter-annotator agreement, and the 'earliest deviating stage' rule likely inflates requirement failures by counting any early omission, even one that never became consequential. The judge also evaluates against the same kind of LLM-synthesized GTs it's being validated with, so anchoring is a real possibility. The paper's own Limitations section concedes that the alignment labels depend on LLM-assisted judging and lists inter-annotator agreement as future work — which is honest, but means the headline claim is not yet fully supported.\n\nNone of this kills the benchmark itself. The resolved rates (31.5% average) stand on executable tests, not on the judge. The clarification and plan coverage scores are also interesting, though they too rely on the LLM judge. What needs fixing is the measurement validity section: publish judge prompts and trajectory pre-processing, release raw run data and per-stage confusion, add inter-annotator statistics, and ideally run an ablation with a different judge or no GTs. That's a revision, not a rewrite.\n\nBottom line: this deserves a serious referee and should go to review, but with a request for major revision on the stage-attribution validation. I'd bring it to a reading group and would probably cite it for the benchmark artifacts, with a caveat about the diagnostics.","headline":"A genuinely useful benchmark for coding agents, but the headline bottleneck claim rests on an LLM judge whose validation is too thin to carry it.","tokens_in":13220,"tokens_out":2390,"would_cite":true,"duration_ms":20107,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces SWE-RPG, a repository-level benchmark that scores coding agents on requirement clarification, planning, and code generation, and finds the main failure stage is recovery of implicit requirements, with average…","keywords":["coding agents","repository-level benchmark","requirement clarification","implementation planning","code generation","implicit requirements","stage-aware evaluation","SWE-RPG"],"falsifier":"A second annotation team, independent of the paper's authors, could reconstruct Clarification and Plan ground truths and stage labels for a stratified sample of the 163 tasks and rerun the judge with only those references; if the dominant failure stage shifts from requirement to planning or code generation, the bottleneck conclusion is a measurement artifact. A lighter check is to compare the reported 92% judge-human agreement on stage labels with an inter-annotator agreement for the reference construction itself, which the paper does not report.","tokens_in":12051,"feed_emoji":"🤖","tokens_out":8239,"duration_ms":67900,"temperature":0.7,"pith_summary":"This paper introduces SWE-RPG, a repository-level benchmark that evaluates coding agents not only on whether their final patch passes tests, but also on the two reasoning stages before code: recovering the request's implicit requirements and forming an implementation plan. The central claim is that this design makes it possible to attribute an unsuccessful run to a specific stage, and the paper's measurements on 163 real issue tasks across 31 Python and Java repositories find that agents resolve only 31.5% of tasks on average, with implicit-requirement recovery the dominant failure stage in 24.5% to 46.0% of runs. The reason this would matter is that current benchmarks typically report only pass or fail, which cannot show whether an agent misunderstood the request, planned poorly, or wrote faulty code; a benchmark with validated intermediate references can point developers toward the capability that actually needs work.","feed_headline":"New benchmark: coding agents solve only 31.5% of tasks","feed_subtitle":"Stage-by-stage diagnosis shows unstated requirements, not code quality, explain most failures.","key_machinery":"The machinery that carries the argument is the stage-attributed diagnosis protocol built on two kinds of validated ground-truth references. Clarification GTs state, in question-answer form, the implementation-critical requirements left implicit in the original request, organized into six categories: functional intent, business semantics, technical context, interface and protocol specifications, code structure and conventions, and data-structure semantics. Plan GTs give ordered, actionable steps whose reproducibility is verified by generating a patch from each step and checking semantic equivalence with the corresponding gold subpatch. An LLM judge then compares each agent trajectory with these references to measure pre-code coverage and to attribute unresolved runs to requirement, planning, code-generation, or other stages, while executable fail-to-pass and pass-to-pass tests from the merged pull request serve as the final correctness oracle.","core_discovery":"SWE-RPG is presented as the first repository-level benchmark to jointly evaluate requirement clarification, implementation planning, and code generation. Each of its 163 tasks supplies an executable patch oracle plus two validated intermediate references: Clarification ground truths, which spell out implementation-critical implicit requirements in six practitioner-derived categories, and Plan ground truths, which provide step-level implementation guidance checked for plan-to-code reproducibility. The evaluation protocol aligns each agent trajectory with these references and assigns every unresolved run to the earliest deviating stage using an LLM judge. On this benchmark, three agent frameworks paired with six language models resolve 31.5% of tasks on average; stage attribution identifies implicit-requirement recovery as the main bottleneck (24.5% to 46.0% of runs), ahead of code generation (7.4% to 37.4%) and planning (5.5% to 17.8%).","pith_inferences":["Editorial inference: a direct test of the bottleneck claim is to supply the Clarification GTs to the same agents at test time and measure the resolved rate; the paper does not run this condition, and if the rate does not rise, requirement recovery is necessary but not sufficient.","Editorial inference: if implicit-requirement recovery is the main bottleneck, trajectory-level signals should show it: agents that spend more effort on repository exploration or clarifying questions before editing should correlate with higher resolve rates on the released runs.","Editorial inference: the six-category clarification taxonomy could be reused as a standalone rubric for requirement-elicitation systems, turning clarification quality into a per-category score; that use is beyond the paper's stated scope."],"forward_implications":["If the benchmark is right, an end-to-end resolve rate alone is a misleading summary: similar resolve rates can accompany very different stage-level failure profiles, so intermediate references are needed to know what to improve.","The dominant failure stage, missed implicit requirements, points to a concrete target: agents should be evaluated and improved on recovering unstated constraints before they plan or edit code.","Planning coverage falls off from target location to implementation approach to constraints and validation, so plan quality should be measured on whether constraints and validation obligations are captured, not just where to edit.","The validated Clarification and Plan references can support more than scoring: they can serve as few-shot demonstrations or training data for the clarification and planning stages themselves."],"supporting_citations":[{"why":"Establishes the executable patch evaluation protocol (fail-to-pass and pass-to-pass tests and resolve rate) that SWE-RPG adopts as its correctness oracle.","marker":"Jimenez et al. 2024"},{"why":"Supplies the agentic build pipeline for constructing per-task Docker environments at base commits, extended with expert-guided recovery.","marker":"Zhang et al. 2025"},{"why":"Closest concurrent work on intermediate reasoning; SWE-RPG distinguishes itself by adding implicit-requirement references and plan-to-code reproducibility.","marker":"Liu et al. 2026"},{"why":"Closest work on clarification in dialogue; SWE-RPG contrasts its naturally mined implicit requirements with that benchmark's synthetic omissions.","marker":"King and Flanigan 2026"},{"why":"Provides the notion of repository-grounded implementation plans that SWE-RPG's Plan GTs are designed to capture.","marker":"Bairi et al. 2024"},{"why":"Grounds the claim that real requirements are continually clarified and that implicit constraints matter in software development.","marker":"Knauss et al. 2015"}],"fun_headline_variants":["Coding agents resolve only 31.5% of tasks; implicit requirements are main failure","SWE-RPG: agents resolve only 31.5%; implicit requirements are main failure","Only 31.5% of coding tasks resolved; implicit requirements trip up agents","SWE-RPG: only 31.5% solved, unstated requirements are main bottleneck"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that implicit requirements are the main bottleneck depends on the accuracy and neutrality of the LLM-generated Clarification and Plan ground truths and of the LLM judge's stage attributions; if those references misstate what a task truly requires or the judge systematically favors one failure stage, the bottleneck ranking is an artifact of measurement rather than a property of the agents.","fun_headline_variants_meta":{"raw":{"variants":["Coding agents resolve only 31.5% of tasks; implicit requirements are main failure","SWE-RPG: agents resolve only 31.5%; implicit requirements are main failure","Only 31.5% of coding tasks resolved; implicit requirements trip up agents","SWE-RPG: only 31.5% solved, unstated requirements are main bottleneck"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001545,"raw_usage":{"total_tokens":6233,"prompt_tokens":1053,"completion_tokens":5180,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":669,"completion_tokens_details":{"reasoning_tokens":5086}},"tokens_in":669,"tokens_out":5180,"duration_ms":29744,"temperature":1.0,"reasoning_tokens":5086,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:09:04.686564+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A second annotation team, independent of the paper's authors, could reconstruct Clarification and Plan ground truths and stage labels for a stratified sample of the 163 tasks and rerun the judge with only those references; if the dominant failure stage shifts from requirement to planning or code generation, the bottleneck conclusion is a measurement artifact. A lighter check is to compare the reported 92% judge-human agreement on stage labels with an inter-annotator agreement for the reference construction itself, which the paper does not report.","supporting_citations":[{"cited_title":"and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir , title =","cited_arxiv_id":null,"evidence_quote":"Establishes the executable patch evaluation protocol (fail-to-pass and pass-to-pass tests and resolve rate) that SWE-RPG adopts as its correctness oracle."},{"cited_title":"C., Vageesh and Iyer, Arun and Parthasarathy, Suresh and Rajamani, Sriram and Ashok, B","cited_arxiv_id":null,"evidence_quote":"Provides the notion of repository-grounded implementation plans that SWE-RPG's Plan GTs are designed to capture."},{"cited_title":"Requirements Engineering , volume =","cited_arxiv_id":null,"evidence_quote":"Grounds the claim that real requirements are continually clarified and that implicit constraints matter in software development."}],"review_version":1}