{"id":"d301c816-0db6-4d0b-bb83-dad35f54b53a","arxiv_id":"2505.13008","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"AdverIntent-Agent infers multiple adversarial program intents, generates tests for each, and produces patches, correctly repairing 77 Defects4J and 105 HumanEval-Java bugs.","lead":"This paper presents a multi-agent system that repairs software bugs by asking a language model to infer several competing 'program intents,' generate tests for each, and patch the code to satisfy them. The system repairs 77 bugs in Defects4J 2.0 and 105 in HumanEval-Java, beating several prior automated repair tools.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Correctness labels for non-exact-match patches rely on the same LLM-generated tests that guided patch generation; if those labels are inflated, the 77/105 headline counts and the outperformance claim lose support.","rationale":"The reader's weakest assumption is intent coverage, but that is not the most load-bearing condition: RQ2 reports 81.7% coverage while the realized repair rate is only 9.2% of Defects4J bugs, so even a large drop in coverage would barely move the headline. The method's success could come from broad patch sampling, better fault localization, or test guidance even when intents are imperfect. The truly load-bearing condition is that the counted correct patches really are correct. The labeling pipeline uses the same LLM and the same generated tests that produced the patches, and the paper does not quantify how much of the final count comes from the weak LLM-based criterion. This is a concrete, checkable vulnerability: if the labels are inflated, the empirical claim collapses because the comparison margins are small (77 vs 74 on Defects4J realistic FL). The paper's manual review is a legitimate mitigation, but without the per-patch breakdown and independent agreement, a reader cannot tell whether the mitigation worked. I therefore keep the verdict conditional on artifact release and independent re-labeling, agreeing with the reader's overall CONDITIONAL verdict but for a different primary reason.","tokens_in":22546,"tokens_out":5916,"duration_ms":62774,"concrete_test":"Release a per-bug ledger for all plausible patches on both benchmarks, classifying each as exact-match, likely-correct, or believed-correct with the two authors' independent labels. Then have two independent evaluators, blind to the tool's labels, re-classify every non-exact-match plausible patch as correct or overfitting against the developer patch. If the independently confirmed correct count for Defects4J realistic FL falls below 74 (ITER's reported count) or for HumanEval-Java below 102 (Cigar's count), the outperformance claim fails as reported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is empirical, and its dependent variable is the set of patches labeled 'correct' in Table 3. Section 4.1.2 defines correct via three routes: exact string match to the developer patch; LLM assessment of whether a plausible patch passes LLM-generated tests derived from an inferred intent; and manual semantic-equivalence review by two authors. The second route is close to circular: Agenttest creates tests from an inferred intent, Agentrepair is prompted to satisfy them, and then a patch that passes 'its intended test case' is called likely-correct. The paper does not report how many of the 77 and 105 counted patches are exact-match versus likely-correct versus believed-correct, nor the inter-rater agreement or cases where manual review overruled the LLM. Since a bug counts as repaired if any generated patch is labeled correct, small labeling errors can alter the headline. The Defects4J realistic-FL count of 77 is within a few bugs of ITER's 74 and SelfAPR's 67, so a handful of false positives changes the 'outperforms' conclusion. The paper acknowledges LLM test-oracle errors in Section 6 (Threat 5), but the manual mitigation is not quantified. The 81.7% intent-coverage result in RQ2 is not the binding constraint: the method repairs only 9.2% of Defects4J bugs, far below that ceiling, so even perfect coverage would not change the headline; label validity is the load-bearing assumption.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AdverIntent-Agent, a multi-agent LLM-based program repair system. The system uses a reasoning agent to infer multiple deliberately adversarial program intents and locate faulty statements, a test agent to generate tests that differentiate these intents (using the same inputs with different expected outputs), and a repair agent to generate patches that satisfy each inferred intent. The authors evaluate on Defects4J 2.0 and HumanEval-Java, reporting 77 and 105 correct repairs under realistic fault localization, respectively, and claim state-of-the-art performance compared with prior APR tools. The paper also includes an ablation study (RQ2) examining the contribution of adversarial intents to fault localization and patch generation, an analysis (RQ3) of how generated tests filter overfitting patches, and a token-cost analysis (RQ4).","tokens_in":22824,"tokens_out":3143,"duration_ms":34016,"significance":"If the reported results hold, this is a valuable contribution to the APR literature. The paper is one of the first to shift the focus of LLM-based repair from generating diverse patches to generating diverse, adversarial program-intent hypotheses, using tests as a way to measure the degree of adversarial disagreement. The evaluation on two standard benchmarks with multiple baselines is appropriate in scope, and the authors make several praiseworthy efforts: they use exact-match and manual semantic-equivalence checks in addition to LLM-based assessment, they explicitly acknowledge data leakage and non-determinism in Section 6, and they state that patches and interaction logs will be publicly released. The idea of supplying developers with inferred intents and tests, rather than only patches, is a novel and potentially useful paradigm. However, the central empirical claim (the 77/105 counts) rests on a patch-correctness labeling procedure that is partly circular, and the reported margins over prior work are small enough that labeling and protocol differences could change the conclusion.","major_comments":[{"comment":"","section":"§4.1.2, Table 3, Table 6"},{"comment":"","section":"§4.2, Table 3"},{"comment":"","section":"§4.3, Table 5, RQ2"},{"comment":"","section":"§4.1, §6, Table 3"}],"minor_comments":[{"comment":"","section":"Table 3"},{"comment":"","section":"§3.2, equation for adversarial score"},{"comment":"","section":"§4.1.2"},{"comment":"","section":"§6"},{"comment":"","section":"§2, Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The central contribution is interesting, but the empirical case is fragile because of the circularity in correctness labeling and the small margins over comparable baselines. The paper would be substantially stronger if the authors provided a breakdown of the headline counts by correctness route, a controlled re-run of key baselines, and a per-bug attribution analysis linking successful repairs to aligned intents. I also note that the relationship to SpecRover (Section 5.3) is close in spirit; the novelty claim rests on the 'adversarial' multiplicity of intents, so the authors should be careful to articulate the measurable difference beyond the number of intents."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth your time. It reframes APR around generating multiple adversarial program intents instead of multiple patches, then uses LLM-generated tests to differentiate those intents and to filter overfitting patches during repair, not after. That is a real conceptual shift, and the integration of test generation into the repair loop is new relative to ChatRepair, Cigar, and SpecRover. The reported numbers—77 correct Defects4J 2.0 repairs and 105 on HumanEval-Java under realistic fault localization—are plausible, and the ablation showing adversarial intents help both fault localization (+13.8%) and patch generation (+19.4%) supports the mechanism. The paper also ships interaction logs and acknowledges data leakage and LLM hallucination as threats.\n\nThe soft spots are real but not fatal. First, the evaluation is a single run of a non-deterministic GPT-4o pipeline with no variance or repeated runs, so the outperformance claims come with unknown noise. Second, and more important, the correctness labels for non-exact-match patches combine LLM judgment with LLM-generated tests that were derived from the same inferred intents that guided patch generation. The paper does not report how many of the 77/105 are exact-match vs. “likely-correct” vs. “believed-correct,” nor the inter-rater agreement for the manual review. Since the Defects4J count of 77 sits within a few bugs of ITER's 74 and SelfAPR's 67, a handful of inflated labels could flip the “outperforms” conclusion. The manual review by two authors mitigates this, but it is not quantified, and the stress-test note is right that label validity is the load-bearing assumption rather than intent coverage (81.7% coverage is not the bottleneck).\n\nI disagree with the skeptic's implied circularity as a fatal flaw. The manual semantic-equivalence review is a standard APR practice, and the paper makes patches public. Still, the authors should break down the correctness counts by route and report agreement before I'd trust the headline numbers as SOTA. Also, bypassing a direct empirical comparison with SpecRover—the closest intent-based baseline—makes the relative novelty argument less sharp.\n\nWho is this for? Anyone working on LLM-based APR or agent-based code repair. It deserves a serious referee: the framing is original, the eval is on standard benchmarks, and the weaknesses are addressable in revision. I would not desk-reject it.\n\nRecommendation: send to peer review, but ask for repeated runs, a correctness-label breakdown, and a SpecRover comparison.","headline":"AdverIntent-Agent is a genuine conceptual step for LLM repair—adversarial intents plus in-loop test generation—but the headline counts rest on a single non-deterministic run and an under-reported correctness-label breakdown.","tokens_in":23383,"tokens_out":2338,"would_cite":true,"duration_ms":21076,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that automated repair succeeds by adversarially inferring several program intents and generating intent-specific tests, correctly repairing 77 Defects4J 2.0 bugs and 105 HumanEval-Java bugs under realistic fault…","keywords":["automated program repair","program intent inference","adversarial reasoning","LLM multi-agent system","test generation","overfitting patch","Defects4J","HumanEval-Java"],"falsifier":"Run each generated adversarial test against the ground-truth fixed program for a random sample of repaired bugs and count how many assert the wrong expected output; a material share of wrong assertions would mean the tests can reject correct patches, undermining the correctness counts. Alternatively, measure intent-alignment coverage on a fresh benchmark and check whether it stays near the reported 81.7 percent, since that coverage is the method's upper bound.","tokens_in":22328,"feed_emoji":"🔧","tokens_out":6565,"duration_ms":58596,"temperature":0.7,"pith_summary":"The paper tries to establish that automated program repair improves when the repair system first reasons about what the developer intended the function to do, rather than only transforming code until the given tests pass. It proposes AdverIntent-Agent, a three-agent LLM system that infers several deliberately different intended behaviors for a buggy function, generates test cases that force those intents to predict different outputs on the same inputs, and then produces patches for each intent. The central claim is that this adversarial spread raises the probability that at least one inferred intent matches the true developer intent, yielding more correct repairs and filtering out patches that merely overfit the supplied tests. Under realistic fault localization, the system correctly repairs 77 of 835 bugs in Defects4J 2.0 and 105 of 164 bugs in HumanEval-Java, more than the compared tools.","feed_headline":"Competing-intent reasoning fixes 77 real Java bugs","feed_subtitle":"A three-agent system infers conflicting program intents, tests each, and beats prior tools on two repair benchmarks.","key_machinery":"The central object is the adversarial program intent: a natural-language specification of a function's expected behavior that is intentionally constructed to conflict with previously inferred intents. The paper measures the conflict with an adversarial score, defined as the fraction of oracle tests (same inputs, intent-dependent expected outputs) on which two intents disagree, and requires each new intent to score above $100\\%/K$ with the first intent. The second load-bearing piece is dynamic precise prompting in the repair agent, which first asks for the top three root causes of the bug under one intent and then issues a separate patch-generation prompt per root cause. Together these mechanisms convert the untestable question “what did the developer mean?” into a space of testable hypotheses, and they force the patch pool to be diverse by construction.","core_discovery":"The paper's central claim is that intent diversity, made concrete through adversarial test oracles, is a repair mechanism in its own right. For a buggy function, the reasoning agent produces an initial natural-language statement of expected behavior and then, prompted with “what if the previous intents are incorrect,” produces two more intents that are deliberately distinct. The test agent turns each intent into executable tests, reusing the same inputs but changing expected outputs per intent, and measures an adversarial score as the fraction of tests whose expected outputs differ between intents; intents below a threshold of 100 percent divided by K are regenerated. The repair agent then asks for the top three root causes consistent with each intent and generates a patch per root cause, accepting only patches that pass both the original tests and the intent-specific adversarial tests. The paper reports that this pipeline correctly repaired 77 Defects4J 2.0 bugs and 105 HumanEval-Java bugs under realistic fault localization, and that on 300 sampled Defects4J bugs at least one of the three inferred intents was judged aligned with the ground-truth intent in 81.7 percent of cases.","pith_inferences":["The method's ceiling is set by intent coverage: if the LLM never proposes the true intent among the K candidates, a correct patch cannot emerge through the intended mechanism, so any way to widen or sharpen the intent search (better prompts, retrieval from issue reports, more candidates) should directly raise the repair ceiling.","The adversarial-score threshold is a diversity heuristic, not a correctness guarantee; one could test whether requiring higher pairwise disagreement between intents actually increases correct-repair yield or instead pushes the model toward implausible intents.","Because the pipeline outputs intent descriptions alongside patches, it offers a natural experiment on developer acceptance: if developers can reliably select the aligned intent, then intent-selection accuracy could become a repair metric in its own right.","The same three-agent loop should transfer to other languages and defect types, but the bottleneck will likely be oracle quality in the generated tests, since incorrect expected outputs can reject correct patches; measuring oracle accuracy on a held-out set of ground-truth fixes would quantify that risk."],"forward_implications":["With intent inference built into repair, correct repairs no longer require a single lucky first patch: the paper's alignment study shows three intents cover the true intent in 81.7% of sampled bugs versus 62.0% for the first intent alone.","Generated adversarial tests act as a filter during repair rather than after it, and in the evaluation they removed likely-overfitting patches for 12 Defects4J bugs and 7 HumanEval-Java bugs.","Adversarial intent exploration also helps locate the bug, improving fault-localization precision by 13.8% and patch-generation success by 19.4% in the ablation study.","The developer-facing output changes from a single patch to a set of inferred intents, tests, and patches, so a human can judge intent in natural language instead of reading a code diff only."],"supporting_citations":[{"why":"Supplies the Defects4J 2.0 benchmark of 835 real-world Java bugs used for the main effectiveness comparison.","marker":"[22]"},{"why":"Supplies the HumanEval-Java benchmark of 164 Java bugs used for the second effectiveness comparison.","marker":"[19]"},{"why":"ChatRepair, the conversation-based repair baseline the approach is compared against and reimplemented for a GPT-4 baseline.","marker":"[62]"},{"why":"RepairAgent, the agent-based repair baseline whose perfect-fault-localization results frame the comparison.","marker":"[4]"},{"why":"Cigar, the cost-efficient conversation-based baseline used in the token-cost comparison.","marker":"[15]"},{"why":"ContrastRepair, a conversation-based baseline using contrastive test pairs, included in the main comparison.","marker":"[24]"},{"why":"ITER, the iterative neural repair baseline included in Table 3 and cited to justify multi-location fault localization.","marker":"[77]"},{"why":"SelfAPR, a fine-tuning-based baseline included in the main comparison.","marker":"[74]"}],"fun_headline_variants":["Adversarial intents fix 77 real Java bugs","Intent diversity repairs bugs via test oracles","Multi-agent reasoning: from patches to intents","Infer conflicting intents to beat overfit patches","When intents clash, patches align with reality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that at least one of the few LLM-inferred adversarial intents matches the developer's true intent: the paper's own RQ2 finds this in 81.7% of 300 sampled Defects4J bugs, leaving 18.3% where the intent-driven pipeline cannot produce a correct patch through its intended mechanism.","fun_headline_variants_meta":{"raw":{"variants":["Adversarial intents fix 77 real Java bugs","Intent diversity repairs bugs via test oracles","Multi-agent reasoning: from patches to intents","Infer conflicting intents to beat overfit patches","When intents clash, patches align with reality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000398,"raw_usage":{"total_tokens":2137,"prompt_tokens":1058,"completion_tokens":1079,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":674,"completion_tokens_details":{"reasoning_tokens":1006}},"tokens_in":674,"tokens_out":1079,"duration_ms":11281,"temperature":1.0,"reasoning_tokens":1006,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:21:03.408968+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run each generated adversarial test against the ground-truth fixed program for a random sample of repaired bugs and count how many assert the wrong expected output; a material share of wrong assertions would mean the tests can reject correct patches, undermining the correctness counts. Alternatively, measure intent-alignment coverage on a fresh benchmark and check whether it stays near the reported 81.7 percent, since that coverage is the method's upper bound.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Defects4J 2.0 benchmark of 835 real-world Java bugs used for the main effectiveness comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ChatRepair, the conversation-based repair baseline the approach is compared against and reimplemented for a GPT-4 baseline."},{"cited_title":"SelfAPR: Self-supervised Program Repair with Test Execution Diagnostics","cited_arxiv_id":"2203.12755","evidence_quote":"SelfAPR, a fine-tuning-based baseline included in the main comparison."}],"review_version":1}