{"id":"ac9765b2-8b81-47ab-95f5-803be8ae8fc4","arxiv_id":"2506.16650","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SemAgent, a workflow-based repair agent using execution, issue, and code semantics, reports solving 134 of 300 SWE-bench Lite issues (44.66%).","lead":"This paper introduces SemAgent, a pipeline that uses execution traces, issue understanding, and code-flow analysis to guide an AI model in fixing software bugs. It reports solving 44.66% of 300 SWE-bench Lite issues, which the authors say beats every other workflow-based repair method.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central 44.66% claim is not evaluable because SemAgent receives preloaded Amazon Q reproduction tests that baseline methods did not; the abstract's superiority claim requires controlled re-runs.","rationale":"The reader identified the preloaded reproduction tests and the use of externally cited baselines as the weakest assumption. My stress-test agrees: this is the single most load-bearing concern for the abstract's central claim that SemAgent beats all other workflow-based approaches on SWE-bench Lite. If the preloaded tests encode expected behavior, the comparison is not controlled, and the 44.66% number cannot be attributed to SemAgent's semantic reasoning. The concern is concrete and testable: re-run baselines with the same preloaded tests, or re-run SemAgent without them. There is also an internal tension in Section 5.2 between claiming no test-specific hints and accepting preloaded reproduction tests as input. The paper's ablation tables and the 51.33% multi-agent union are secondary concerns; the primary empirical claim is the one threatened by the preloaded tests. Since the reader's verdict was CONDITIONAL, and this concern supports that conditional status rather than overturning it, I recommend UNCHANGED.","tokens_in":13901,"tokens_out":3909,"duration_ms":38913,"concrete_test":"Re-run the 300 SWE-bench Lite issues with Agentless (or SpecRover) under the exact SemAgent input protocol: give the baseline the issue description, codebase, and the same preloaded Amazon Q reproduction tests, using the same single temperature-0 run and the same evaluation harness. If Agentless with the preloaded tests reaches or exceeds 44.66%, the claimed superiority is not established. Alternatively, re-run SemAgent without the preloaded tests (using only its own reproducer generation for execution semantics) and compare the solve rate; if it drops materially below 44.66%, the headline number is inflated by the externally supplied reproduction tests.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result, 44.66% on SWE-bench Lite with the claim of beating all workflow-based methods, depends on a fair comparison against Agentless (40.67%), SpecRover (37%), and OpenHands (41.67%). Section 5.2 states: 'SemAgent takes as input the issue description, codebase, and preloaded Amazon Q Developer reproduction tests (AWS, 2025).' Section 5.3 reports that the baseline numbers are taken from prior papers, and those methods were not run with these preloaded tests. Providing executable reproduction tests as input gives SemAgent information about the expected behavior that is not available to the cited baselines. The paper's Section 4.1 explicitly uses reproducer crash traces and spectrum-based fault localization for bug localization, so the preloaded tests directly feed into the localization stage and can also be used to validate candidate patches. This is a confound: the 44.66% may reflect the extra input rather than the semantics-aware pipeline itself. The tension is visible in Section 5.2, which asserts SemAgent 'uses no test-specific metadata or hints' while accepting preloaded tests as an input. If the preloaded tests encode the issue's expected behavior (which a reproduction test by definition does, since it should fail before the fix and pass after), the superiority claim over methods evaluated without such tests is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SemAgent, a workflow-based automated program repair agent that combines execution semantics (reproducer stack traces, execution traces, spectrum-based fault localization), issue semantics (an LLM-generated generalization of the issue statement), and code semantics (structured execution flows and step-level code mapping) to produce patches for SWE-bench Lite issues. The pipeline first localizes the bug using execution-semantics evidence, generates an initial patch via an APR workflow, then refines it through a repair stage that produces fine-grained fixes, a reviewer agent that filters relevant fixes, and a patch aggregator that merges and validates the final patch. The authors report a solve rate of 44.66% (134/300) on SWE-bench Lite, claim this is the highest among workflow-based methods, and present a multi-agent extension estimate of 51.33% (154/300) based on the union of ablated configurations. The paper includes ablations of the repair stage, issue semantics, reviewer agent, and execution-semantics-based localization.","tokens_in":14188,"tokens_out":3031,"duration_ms":30205,"significance":"If the 44.66% SWE-bench Lite result is established under a fair comparison, this is a meaningful advance for deterministic, workflow-based APR: it uses an out-of-the-box LLM with zero decoding temperature, requires no fine-tuning, and provides full prompts in the appendix, which supports reproducibility of the pipeline design. The paper explicitly targets the known failure mode of hyper-localized patches and provides a motivating example (astropy-14635) where SemAgent produces a more complete fix than typical overfitted patches. The ablation structure is sensible and the components are clearly described. However, the headline comparison is currently undermined by an evaluation-setup confound (preloaded reproduction tests provided only to SemAgent and not to baselines), and the multi-agent 51.33% figure is presented as a result despite being an untested union of separate runs. The issue-semantics ablation also includes the motivating example used to design the prompt, introducing a circularity that weakens the causal claim about issue semantics.","major_comments":[{"comment":"The headline comparison against Agentless (40.67%), SpecRover (37%), and OpenHands (41.67%) is confounded by the preloaded Amazon Q Developer reproduction tests. Section 5.2 states that SemAgent takes as input the issue description, codebase, and preloaded Amazon Q Developer reproduction tests, while Section 5.3 reports that the baseline numbers are taken from prior papers and the baselines were not re-run with these preloaded tests. Since Section 4.1 uses reproducer crash traces and spectrum-based fault localization, these tests can directly influence localization and patch validation, providing SemAgent with information about expected behavior that the cited baselines did not receive. The abstract's claim of beating all workflow-based approaches therefore is not supported by the current experimental design. The authors should either re-run the baseline methods with the same preloaded tests, or ablate SemAgent without the preloaded tests, and report both comparisons.","section":"§5.2, §5.3, Table 1"},{"comment":"The issue-semantics ablation is circular because it includes the very example used to design the issue-semantics prompt. Section 3.1 introduces astropy-14635 as the motivating example underlying the design, and Section 6.2 states that Table 3 evaluates issue semantics on '49 randomly selected low-cost issues along with the motivating example.' Including this example in the 50-issue ablation means the prompt was effectively tuned on a test point, which can inflate the observed 60% vs. 50% difference and undermines the claim that issue semantics provides a causal improvement. The ablation should be re-run on the 49 randomly selected issues alone, excluding astropy-14635, to establish that the benefit generalizes.","section":"§6.2, Table 3 (RQ2)"},{"comment":"The reported 51.33% for SemAgent-Multi is not an evaluated system; it is the size of the union of issues solved by different ablation configurations, as shown in Figure 3. Table 1 lists SemAgent-Multi as a workflow-based approach with a 51.33% solve rate, but Section 6.1 only says that 'extending our aggregator to a multi-agent setup, we could potentially solve up to 154 issues.' Presenting this untested upper bound in the main leaderboard table as if it were a measured result is misleading. The multi-agent configuration should be actually run and evaluated, or the table should clearly label the 154-issue figure as an upper-bound estimate rather than a solved rate.","section":"§6.1, Table 1, Figure 3"},{"comment":"The evaluation is a single run with no released patches (Section 6.1: 'All results are from a single run'), and the full system relies on a proprietary model (Claude Sonnet 3.7). While temperature zero and a deterministic workflow reduce variance, LLM-based steps can still vary across runs, and the absence of any variance estimate or multiple-run data makes the 44.66% point estimate hard to assess for statistical significance. The authors should provide at least a small multiple-run study on a random subset (e.g., 50 issues) with confidence intervals, and release the generated patches for the 300 issues to enable independent verification.","section":"§6.1, §8"}],"minor_comments":[{"comment":"The text refers to 'SpecRover (Zhang et al., 2024)' but the reference list entry is Ruan et al. (2024); the citation should be corrected.","section":"§5.3"},{"comment":"Table 3 is titled 'Random Set of 50 SWEBench Issues', but the text says the set is '49 randomly selected low-cost issues along with the motivating example'; the title and text should be made consistent, and the inclusion of a non-random example should be disclosed in the table itself.","section":"§6.2, Table 3"},{"comment":"The caption 'Overlap of Ablations - 154 issues resolved in total' should clarify that this is the union of issues resolved by different ablation variants, not by a single system, to avoid confusion with the table entry.","section":"Figure 3 caption"},{"comment":"There are typos such as 'acctute' and 'repostiory-level' in the first paragraph of Related Work; the text should be proofread.","section":"§2"},{"comment":"The statement that SemAgent 'uses no test-specific metadata or hints' is in tension with the preloaded reproduction tests being an input; the authors should clarify the intended distinction between 'test-specific metadata' and 'reproduction tests' to avoid an apparent contradiction.","section":"§5.2"},{"comment":"The limitation that the current implementation is restricted to single-file fixes is acknowledged, but it would be helpful to report how many of the 300 SWE-bench Lite issues involve multi-file gold patches, since the headline result may partly reflect the benchmark composition.","section":"§8"}],"recommendation":"major_revision","confidential_remarks":"The paper is well written and the pipeline is thoughtfully designed, but the evaluation currently overclaims in two load-bearing ways: the baseline comparison is confounded by the preloaded reproduction tests, and the SemAgent-Multi 51.33% figure is an untested union rather than a measured result. The issue-semantics ablation also includes the design example, which is a subtle but real circularity. These are fixable with controlled re-runs, and the authors have the infrastructure to do them; this is a major-revision case, not a reject."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nTwo things to know before you spend time on this paper. First, the pipeline is genuinely well designed: execution semantics (reproducer refinement, trace-based file ranking, SBFL) feed a repair stage that uses issue-semantics abstraction and step-level code-semantics mapping, and a reviewer agent filters the candidate fixes. The internal ablations tell a clean story—baseline 37%, +5% with the repair stage, +2.66% with the reviewer—and the prompts are in the appendix, so reimplementation is feasible. Credit where it is due: this is a coherent, serious architecture.\n\nSecond, the headline number is not currently a fair claim. SemAgent is run with preloaded Amazon Q reproduction tests (Section 5.2), while the baselines in Section 5.3 are taken from prior papers and were not given those tests. Because a reproduction test fails before the fix and passes after, it encodes expected behavior and feeds localization and validation. The paper says SemAgent \"uses no test-specific metadata or hints,\" which is hard to square with using reproduction tests as an input. The abstract's claim of beating all workflow-based approaches is therefore not established by this evaluation.\n\nOther soft spots are less severe but still real. The 51.33% \"SemAgent-Multi\" in Table 1 is not an executed system; it is the union of issues resolved across different ablations (Figure 3 says \"could potentially solve up to 154\"). Presenting it as a leaderboard number is misleading. The issue-semantics ablation runs on 50 issues and includes the motivating example astropy-14635, so that validation is not clean. No code or patches are released, and the results are a single run with a proprietary model, so independent checking is impossible.\n\nNone of this kills the core idea. The internal ablations suggest that semantics-aware repair does help, and the evaluation problems are fixable: re-run baselines with the same reproducer input (or re-run SemAgent without preloaded tests), report the multi-agent result as a potential upper bound, and clean the ablation set. This paper deserves a serious referee, but it needs major revision before the claims are supportable.\n\nRegards.","headline":"Well-built pipeline with plausible internal ablations, but the leaderboard claim is confounded by preloaded reproduction tests and the 51.33% multi-agent number is an unrun union.","tokens_in":14746,"tokens_out":3071,"would_cite":false,"duration_ms":32731,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SemAgent, a deterministic workflow-based repair agent, argues that execution, issue, and code semantics—not more autonomy—produce complete patches, reporting 134 of 300 SWE-Bench Lite issues solved (44.66%), the highest among…","keywords":["automated program repair","repository-level bug fixing","LLM agents","workflow-based repair","issue semantics","code semantics","execution semantics","SWE-bench Lite"],"falsifier":"Re-run the published workflow baselines with the same preloaded reproduction tests and the same temperature-zero settings, and re-run SemAgent without those preloaded tests. If any baseline reaches 134 of 300 issues, or if SemAgent drops below 40.67% when the preloaded tests are removed, the paper's claim that it is the best workflow-based method on this benchmark would be contradicted.","tokens_in":13644,"feed_emoji":"🐛","tokens_out":11928,"duration_ms":106011,"temperature":0.7,"pith_summary":"Automated repair agents often stare at the most suspicious lines and patch only those, producing fixes that are overfitted to the issue's example or that miss consistency and edge cases. This paper argues the remedy is to make the pipeline semantics-aware: use execution behavior to localize the bug, generalize the issue into abstract directions, and trace the code's execution flows before generating fixes. The result is SemAgent, a deterministic workflow that solves 134 of 300 issues on the SWE-Bench Lite benchmark (44.66%), which the authors report as the highest among workflow-based methods and a 7.66-point gain over their SpecRover-style baseline. If the claim holds, it would mean that structured, interpretable workflows can outperform open-ended agentic systems on repository-level repair without fine-tuning.","feed_headline":"Semantics-aware repair agent solves 44.66% of SWE-Bench Lite issues","feed_subtitle":"A fixed workflow using issue, code, and execution semantics beats open-ended agentic systems on 300 GitHub issues","key_machinery":"The load-bearing mechanism is a three-channel semantics model feeding a two-stage repair-and-review architecture. Execution semantics triangulate the buggy region through reproducer stack traces, execution-trace file lists, and SBFL suspiciousness rankings. Issue semantics is a generalized '<directions>' block extracted by prompting the LLM to abstract past the specific failing example. Code semantics is built from execution flows—high-level paths through the file, decomposed into steps and mapped to concrete code snippets through structured JSON. These feed a repair stage that proposes fine-grained fixes and a reviewer stage that filters to issue-relevant fixes; an aggregator then deduplicates, merges, and refines via an execution loop. In the paper's ablations, adding the repair stage raises the baseline from 37% to 42%, the reviewer adds another 2.66 points, and issue semantics alone accounts for 10 points on a 50-issue subset.","core_discovery":"SemAgent is defined as a function $F(I, D, T) \\to p$ that turns an issue, a codebase, and regression tests into a final patch. It first localizes with execution semantics—stack traces from an iteratively refined reproducer, files appearing in execution traces, and spectrum-based fault localization—then runs a standard patch generation, validation, and selection cycle to get an initial patch. The repair stage abstracts the issue into generalized directions (issue semantics), decomposes the buggy file's execution flows into natural-language steps mapped to code snippets (code semantics), and proposes fine-grained fixes. A reviewer agent filters fixes by relevance, and an aggregator merges them into a final patch. On 300 SWE-Bench Lite issues the paper reports 134 solved (44.66%), 7.66 points above its baseline, and attributes the gain to semantics plus the reviewer rather than to a stronger model.","pith_inferences":["Beyond the paper: a clean test of the semantics thesis would run SemAgent with and without the preloaded reproduction tests; if the gains vanish, the contribution is test provisioning rather than semantic understanding.","Beyond the paper: because the method is deterministic and caches repair-stage results, its cost model (average $6.9 per issue) suggests a practical deployment could use cheaper models for localization and reserve the expensive model for the repair and review stages.","Beyond the paper: the single-file scope means multi-file consistency bugs are out of reach; extending the flow-to-snippet mapping to a repository-level call graph would be a direct testable extension of the code-semantics machinery.","Beyond the paper: the gap between the 44.66% single-run result and the 51.33% multi-agent overlap hints that a selection policy for choosing among candidate semantic fixes could recover most of that gap without new semantic signals."],"forward_implications":["Deterministic workflow-based repair, using an off-the-shelf model, can exceed open-ended agentic systems on the benchmark if the semantic components are the source of the gain.","Adding issue-semantics abstraction to an existing Search-Edit-Test pipeline is a drop-in change that could transfer the method's benefits to other repair systems.","The reviewer stage's 2.66-point contribution suggests that filter-and-merge post-processing is an effective precision lever independent of generation quality.","The reported multi-agent overlap of 154 issues implies that different semantic views produce complementary fixes, so aggregating diverse semantic agents is a path toward higher solve rates without a stronger base model.","Execution semantics improve localization by 3.67 points overall and matter most for repositories with heavy setup, which guides where to invest in reproducer infrastructure."],"supporting_citations":[{"why":"Provides SWE-Bench and the 300-issue SWE-Bench Lite benchmark on which the 44.66% solve rate is measured.","marker":"Jimenez et al., 2024"},{"why":"SpecRover: the architectural baseline whose reproducer generation, context localization, and patching SemAgent reuses, and whose 37% score is the comparison point for the 7.66-point gain.","marker":"Ruan et al., 2024"},{"why":"Agentless: the strongest workflow-based comparison at 40.67%, whose hyper-localized patches SemAgent is designed to improve on.","marker":"Xia et al., 2024"},{"why":"Supplies spectrum-based fault localization, one of the three execution-semantics signals used for bug localization.","marker":"Abreu et al., 2007"},{"why":"Source of the preloaded Amazon Q Developer reproduction tests that SemAgent takes as input; the load-bearing input whose absence in baseline runs is the paper's key evaluation premise.","marker":"AWS, 2025"},{"why":"OpenHands: the agentic system at 41.67% that SemAgent is compared against to show workflow-based methods can compete with agentic ones.","marker":"Wang et al., 2024b"},{"why":"AutoCodeRover: a workflow-based baseline at 30.67%, part of the comparison set for the claim of highest workflow-based performance.","marker":"Zhang et al., 2024"}],"fun_headline_variants":["SemAgent: semantics-aware repair solves 44.66% of SWE-Bench Lite","Semantics-driven repair agent solves 44.66% on SWE-Bench Lite","SemAgent improves SWE-Bench Lite solve rate by 7.66% using semantics","SemAgent leverages semantics to beat workflow-based agents on SWE-Bench Lite"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparison assumes it is fair to give SemAgent preloaded reproduction tests while taking baseline scores from earlier runs that did not have those tests.","fun_headline_variants_meta":{"raw":{"variants":["SemAgent: semantics-aware repair solves 44.66% of SWE-Bench Lite","Semantics-driven repair agent solves 44.66% on SWE-Bench Lite","SemAgent improves SWE-Bench Lite solve rate by 7.66% using semantics","SemAgent leverages semantics to beat workflow-based agents on SWE-Bench Lite"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001367,"raw_usage":{"total_tokens":5596,"prompt_tokens":1054,"completion_tokens":4542,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":670,"completion_tokens_details":{"reasoning_tokens":4451}},"tokens_in":670,"tokens_out":4542,"duration_ms":31242,"temperature":1.0,"reasoning_tokens":4451,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:21:20.328056+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the published workflow baselines with the same preloaded reproduction tests and the same temperature-zero settings, and re-run SemAgent without those preloaded tests. If any baseline reaches 134 of 300 issues, or if SemAgent drops below 40.67% when the preloaded tests are removed, the paper's claim that it is the best workflow-based method on this benchmark would be contradicted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies spectrum-based fault localization, one of the three execution-semantics signals used for bug localization."}],"review_version":2}