{"id":"647470f2-6677-4b0f-b337-77cdc8e2f0c3","arxiv_id":"2506.17772","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLM repair agents fail mostly on type and code-context problems, and a hybrid static-analysis/LLM tool fixes 29 of 127 previously failed type-related patches.","lead":"This paper examines 769 failed software patches produced by seven LLM-based repair agents, then groups the failures into six root-cause categories. It also introduces PAGENT, a static-analysis-assisted rewriter that turns 29 of 127 previously failing type-related patches into passing patches.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 127-patch denominator for the headline 29/127 result is not derivable from the paper's own counts, so the central 22.83% improvement claim is unsupported as reported.","rationale":"I read the paper in good faith: it is an empirical study plus a post-processing tool, and the central claim is a quantitative one — PAGENT fixes 29 of 127 previously failed type-related patches. For that claim to hold, the evaluation set (the 127 patches) must be well-defined and countably consistent with the study's own taxonomy. The paper's internal numbers fail this test: the 127 denominator conflicts with Table 3 (44 for the three named agents), with §5.1's 762/37.34% statement (implying ~284), and with the Conclusion's 9.37% (implying ~310). This is not a matter of external consensus or a missing baseline; it is a direct arithmetic inconsistency in the evidence for the headline rate. The reader's weakest assumption concerned inter-rater reliability of the taxonomy labels. My concern is narrower and more concrete: even if the labels were perfectly reliable, the counts do not add up, so the rate is not trustworthy as reported. I partially agree with the reader because both concerns target the integrity of the evaluation set, but my emphasis is on the observable numerical contradiction rather than the subjective labeling. I do not recommend outright rejection: the underlying approach might still work, and a released artifact or a corrected table could resolve the discrepancy. Since the reader's verdict was already CONDITIONAL and my concern remains addressable, I recommend keeping the verdict UNCHANGED, with the condition that the authors release the 127-patch list and reconcile the denominator across Sections 5 and 8.","tokens_in":16121,"tokens_out":7000,"duration_ms":66905,"concrete_test":"Obtain the list of 127 instance IDs used in RQ1 (or the released repository) and recompute the type-related counts per agent from the raw GPT-4o analysis reports. Specifically, compare the 127 to the sum of Table 3's 'Type & data structure' column for Agentless, Aider, and AutoCodeRover (currently 44). If the 127 list cannot be reproduced, or if the true count is 44/98, rerun PAGENT on the corrected set and recompute the improvement rate. Also verify whether the Conclusion's 9.37% uses a different denominator (≈310) and reconcile §5.1's 762/37.34% statement with these counts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result, 'PAGENT resolved 29 of 127 type-related failed patches (22.83%)' (Abstract; §5.2 RQ1), is not reproducible from the paper's own accounting. Table 3 lists the type-and-data-structure counts for the three evaluated agents (Agentless, Aider, AutoCodeRover) as 17, 14, and 13, summing to 44; across all seven agents the sum is 98. Yet §5.2 uses a denominator of 127 without explaining where those additional 83 patches come from. Section 5.1 states there are '762 patches, representing approximately 37.34% of all failed patch attempts,' which would imply about 284 type-related patches, not 127. The Conclusion (Sec. 8) gives a '9.37% improvement rate on type-related issues' — for 29 fixes that implies a denominator near 310, again not 127. These three numbers (127, ~284, ~310) cannot all describe the same evaluation set. Since the central claim is a rate, an unstable denominator means the reported 22.83% improvement is unsupported as written. The discrepancy also casts doubt on the stability of the 'type-related' labeling pipeline (Sec. 2.3), because the count of type-related patches changes by agent/dataset depending on which section is read. This is an internal consistency problem, not a disagreement with external consensus.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical study of failed patches produced by seven LLM-based issue-resolution agents on 114 SWE-bench Lite issues, presenting a taxonomy of six failure categories. The authors identify insufficient type and data structure handling as a frequent and addressable failure mode, and propose PAGENT, a post-processing agent that combines AST parsing, CFG construction, reaching-definitions analysis, LLM-based type refinement, and mypy-aware rewriting. PAGENT is applied to previously failed type-related patches, and the abstract and RQ1 report that it fixed 29 of 127 such patches, a 22.83% improvement rate, with model-specific gains for Aider, Agentless, and AutoCodeRover. The paper also analyzes the sub-categories of successfully fixed type-related issues.","tokens_in":16414,"tokens_out":3354,"duration_ms":33745,"significance":"If the empirical claims are correct, the work has two useful contributions: a structured taxonomy of failure modes for LLM code agents, and a concrete demonstration that targeted static analysis plus type inference can rescue a meaningful fraction of failed patches. The evaluation uses the external SWE-bench Lite harness through the Moatless platform, which is a strength because the headline result is measured by real test execution rather than by the authors' own proxy. The PAGENT design is clear and the worked examples are instructive. However, the central quantitative claim is currently not reproducible from the paper's own tables and sections, and the evaluation lacks a baseline that separates the contribution of static analysis from that of the LLM rewriting step. These issues must be resolved before the reported improvement rate can be accepted.","major_comments":[{"comment":"The headline result, 29 of 127 type-related failed patches (22.83%), is not derivable from the paper's own counts. Table 3 reports type-and-data-structure counts of 17, 14, and 13 for Agentless, Aider, and AutoCodeRover, respectively; these three sum to 44, and all seven agents sum to 98. Section 5.1 says there are 762 patches representing 37.34% of all failed patch attempts, which would imply roughly 284 type-related patches, not 127. Section 8 reports a 9.37% improvement rate, which for 29 fixes implies a denominator near 310. These numbers cannot all describe the same evaluation set, so the 22.83% rate and the conclusion's 9.37% rate are internally inconsistent. The authors must provide a consistent accounting of the evaluation set and recompute all rates.","section":"Abstract, §5.2, Table 3, §5.1, §8"},{"comment":"The taxonomy that drives the selection of the 127 type-related patches (or whatever the correct denominator is) is produced by GPT-4o plus manual review, but no inter-rater reliability or validation of the labels is reported. Since the central claim depends on correctly identifying which failed patches are type-related, the absence of any agreement metric makes the result vulnerable to categorization bias that Section 6.2 itself acknowledges. I ask for at least a quantified agreement measure on a sample of labels, or an alternative validation of the labeling procedure.","section":"§2.3, §6.2"},{"comment":"There is no baseline or ablation that isolates the contribution of the static-analysis components (AST, CFG, reaching definitions) from the contribution of the LLM-based type refinement and patch rewriting. Without a condition that, for example, feeds the same failed patches to an LLM rewriter without the static-analysis context, or applies static analysis without the LLM refinement, it is not possible to attribute the 29 fixes to PAGENT's claimed novelty. Reporting such an ablation is necessary to support the paper's central design claim.","section":"§4, §5"},{"comment":"The model-specific improvement rates in Table 5 are expressed relative to each agent's total success rate on 300 issues, but PAGENT is applied only to a subset of previously failed patches. The correct denominator for a patch-level improvement rate is the set of type-related failed patches per agent, not the full 300-issue set. As reported, the 'relative gain' conflates patch-level fixes with overall issue-level success rates and is therefore difficult to interpret. Please report, per agent, the number of type-related failed patches and the number of those that PAGENT fixed.","section":"§5.3, Table 5"}],"minor_comments":[{"comment":"The text says PAGENT was applied to type-related failures 'cross all seven agents' but then reports results only for three agents; please clarify whether the evaluation set was the top three agents or all seven, and reconcile this with the abstract's statement about the top three agents.","section":"§5.2"},{"comment":"The sentence 'In total, there are 762 patches, representing approximately 37.34% of all failed patch attempts' is confusing: based on the data collection in §2.2, 769 cases were processed (with 29 excluded), so 762 cannot represent 37.34% of the total. Please clarify what 762 counts and what the 37.34% refers to.","section":"§5.1"},{"comment":"The percentages in Figure 3 do not appear to be directly derivable from the raw counts in Table 3; for example, the type-handling percentage of 37.34% is not consistent with 98 out of the total failures shown in Table 3. Please explain the computation or the relationship between the figure and the table.","section":"§3.7, Table 3"},{"comment":"There are several typos that should be corrected: 'Optinal' should be 'Optional', 'endwith' should be 'endswith', 'Meatless' should be 'Moatless' in §2.2, 'leafing' should be 'leading' in §3.6, and the agent name 'AppMap Naive' is spelled inconsistently as 'Navie' in several places.","section":"§4.5, Table 4"},{"comment":"The paper reports 18 unique issues across 29 total occurrences, but does not state how the overlap across agents is handled when computing the 22.83% improvement rate; please clarify whether the 29 fixes are counted per agent or per unique issue, since the abstract's '29 patches' suggests per-patch counting while RQ3 uses unique issues.","section":"§5.4, Figure 8"}],"recommendation":"major_revision","confidential_remarks":"The core ideas are interesting and the external evaluation harness is a strong point, but the internally inconsistent denominator is a serious reporting defect that affects the paper's headline result. The fix is feasible: the authors need to provide a consistent dataset accounting, validate the taxonomy labels, and add baseline/ablation experiments. This is a major-revision situation rather than a reject, provided the corrected numbers still support the central claim. I would also check whether the 127-patch set includes patches from outside the top three agents, since that may resolve part of the inconsistency."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things worth knowing. The failure taxonomy is the real contribution: six categories, grounded in 769 real patches on SWE-bench Lite, with concrete examples from sympy, scikit-learn, and sphinx. That is useful empirical work. The 29 verified fixes from PAGENT are also real as far as I can tell—they are validated on the SWE-bench test harness. But you cannot trust the 22.83% improvement rate. The denominator is not stable anywhere in the paper. Abstract and RQ1 say 127 type-related patches from the top three agents; Table 3 gives 44 for those three agents and 98 for all seven. Section 5.1 tosses in \"762 patches\" which is not 37.34% of anything, and the conclusion says 9.37% improvement, which implies a denominator around 310. The same 29 fixes cannot be both a 22.83% and a 9.37% improvement. This is not a subtle point; it is the headline number.\n\nOn the taxonomy, I would have liked inter-rater reliability or some validation of the LLM-plus-manual labeling. The paper acknowledges categorization bias in Section 6.2 but does not measure it. On PAGENT, there is no ablation separating static analysis from LLM type inference, so I don't know which component matters. No code or prompts are released, which makes the 29 fixes hard to reproduce independently.\n\nThat said, the paper is not a wreck. The tool design is reasonable—AST/CFG/reaching definitions plus LLM refinement and mypy is a sensible recipe for a post-processor. The 29 fixes show the approach can work on real repositories. The numeric conflicts look like careless reporting rather than fraud, especially since the per-model counts in Table 5 (+11, +10, +8) sum to 29 and are consistent.\n\nWho should read it? People working on LLM-based repair and patch validation. The taxonomy gives a shared vocabulary for failure modes. The PAGENT recipe is worth trying. Point your students at it, but warn them about the numbers.\n\nRecommendation: send it to review, but expect major revision. The authors need to fix the denominator, reconcile the abstract with the conclusion, add taxonomy validation, and release artifacts. With those, this could be a decent contribution. Without them, the headline rate is not defensible.","headline":"The failure taxonomy is a genuine empirical contribution, but the headline 22.83% improvement rate is unsupported by the paper's own numbers; the underlying PAGENT idea still deserves a serious look.","tokens_in":16850,"tokens_out":4018,"would_cite":false,"duration_ms":35332,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A post-processing layer that combines static program analysis with an LLM can convert a meaningful fraction of failed AI-generated patches into passing fixes.","keywords":["automated program repair","LLM-based software agents","failure taxonomy","type inference","static analysis","control flow graph","SWE-bench Lite","patch rewriting"],"falsifier":"Reclassify the 769 failed patches with two independent human annotators who are blind to PAGENT's outcome, then count how many of PAGENT's 29 fixed patches fall outside the type-related label; if more than a handful fall outside, the 22.83% rate reflects label selection rather than type repair.","tokens_in":15929,"feed_emoji":"🐛","tokens_out":6122,"duration_ms":57548,"temperature":0.7,"pith_summary":"This paper asks why LLM-based issue-resolution agents fail and whether those failures can be repaired after the fact. Studying 114 issues from SWE-bench Lite that none of seven agents could resolve, it classifies 769 failed patches into six failure patterns, finding that insufficient data-type and structure handling is one of the most common causes. To exploit that finding, it builds PAGENT, a post-processing layer that statically analyzes the repository around a patch, infers variable types, and rewrites the patch accordingly. On 127 type-related failed patches from three agents, PAGENT turned 29 into passing patches, a 22.83% improvement rate. The result matters because it suggests that a targeted, model-agnostic repair step can salvage a substantial share of failures that otherwise look like general LLM limitations.","feed_headline":"Post-processor converts 29 failed AI patches into passing fixes","feed_subtitle":"A static-analysis-plus-LLM layer raises three repair agents' pass rates on SWE-bench Lite type failures.","key_machinery":"The machinery is PAGENT's four-stage pipeline: a patch analysis module that validates diff format and syntax; a code analysis module that parses the repository into abstract syntax trees, builds control-flow graphs, and runs reaching-definitions analysis to track what values a variable can hold; a type inference system that combines these static results with an LLM's contextual judgment to assign types; and a patch rewriter that regenerates the diff and checks it with a static type checker. The control-flow graph and reaching-definitions analysis do the load-bearing work of overcoming localized patch inspection by tracing type flow across execution paths.","core_discovery":"The central claim is that type-related patch failures form an identifiable, separable failure class, and that a hybrid static-analysis-plus-LLM post-processor can repair roughly one in five of them. The paper states this as 29 resolved patches out of 127 previously failed type-related patches, a 22.83% improvement rate. It also claims the empirical taxonomy identifies six failure patterns—type and data structure handling, shallow code-context understanding, inadequate error handling, limited integration with existing utilities, cross-version compatibility, and performance implications—with type handling and code context among the most frequent.","pith_inferences":["Because the 127-patch test set is defined by the taxonomy labels, an independent re-labeling of the same 769 failures would show whether the 22.83% rate is really about types or about some correlated property of the patches.","The same CFG-plus-LLM pipeline could plausibly be applied to other taxonomy categories, such as missing error handling or overlooked utility functions, but the paper does not test that.","The paper's own unfixed-case analysis suggests the ceiling is structural: even with correct type information, patches fail when fixes require multi-module architectural reasoning or domain-specific API knowledge."],"forward_implications":["A post-processing layer can raise an agent's effective pass rate without retraining the agent, with the reported gains coming from three different base agents whose type-related failures were most frequent.","The failed-patch taxonomy provides a route to build other targeted repair tools, since each category names a specific, localized failure mechanism rather than leaving failures as a general LLM weakness.","Type-related errors involving complex nested data structures are the most tractable for this static-analysis approach, since they made up 44.4% of the 18 unique issues PAGENT fixed.","Agents whose original patches already handle types adequately gain nothing from this repair layer, since four of the seven evaluated agents showed no improvement."],"supporting_citations":[{"why":"Supplies the SWE-bench Lite benchmark, the 114 unresolved issues, and the gold patches used to judge the taxonomy and PAGENT.","marker":"[14]"},{"why":"One of the top three base agents whose failed type-related patches PAGENT was evaluated on.","marker":"[34]"},{"why":"One of the top three base agents whose failed type-related patches PAGENT was evaluated on.","marker":"[29]"},{"why":"One of the top three base agents whose failed type-related patches PAGENT was evaluated on.","marker":"[9]"},{"why":"The evaluation harness used to apply refined patches and run the SWE-bench test suite.","marker":"[24]"},{"why":"Contributes failed patches from two model configurations to the 114-issue corpus and the taxonomy.","marker":"[23]"},{"why":"Contributes failed patches to the corpus and the taxonomy as one of the seven studied agents.","marker":"[32]"},{"why":"Contributes failed patches to the corpus and the taxonomy as one of the seven studied agents.","marker":"[5]"}],"fun_headline_variants":["PAGENT post-processor fixes 29 type-failed AI patches","Type-error fixer repairs 22.8% of failed AI patch attempts","Static analysis plus LLM rescues 29 AI patch failures","AI patches: taxonomy reveals type errors, PAGENT fixes 29"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole 29-fix result rests on the assumption that the LLM-assisted manual review correctly identified which failed patches were type-related, because that label determines both the 127-patch test set and the claimed 22.83% improvement rate.","fun_headline_variants_meta":{"raw":{"variants":["PAGENT post-processor fixes 29 type-failed AI patches","Type-error fixer repairs 22.8% of failed AI patch attempts","Static analysis plus LLM rescues 29 AI patch failures","AI patches: taxonomy reveals type errors, PAGENT fixes 29"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000216,"raw_usage":{"total_tokens":1413,"prompt_tokens":904,"completion_tokens":509,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":431}},"tokens_in":520,"tokens_out":509,"duration_ms":4818,"temperature":1.0,"reasoning_tokens":431,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:01:10.532668+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reclassify the 769 failed patches with two independent human annotators who are blind to PAGENT's outcome, then count how many of PAGENT's 29 fixed patches fall outside the type-related label; if more than a handful fall outside, the 22.83% rate reflects label selection rather than type repair.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the top three base agents whose failed type-related patches PAGENT was evaluated on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The evaluation harness used to apply refined patches and run the SWE-bench test suite."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes failed patches to the corpus and the taxonomy as one of the seven studied agents."}],"review_version":2}