{"id":"65236c34-7203-46ec-8beb-83544d96a45b","arxiv_id":"2608.06410","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Persistently tracking repair issues instead of only candidate performance improves automated full-code AI agent design by 25.2% on average over five benchmarks.","lead":"This paper introduces ADIAS, an automated system that improves AI agents by maintaining a persistent list of unresolved issues and using that list to guide each code repair. Across five interactive benchmarks, ADIAS beat the strongest prior method by 25.2% on average, suggesting that tracking 'what is still broken' is more effective than only tracking how each candidate agent scored.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Persistent-issue mechanism is unverifiable as stated: Section 6 does not measure diagnosis or issue-association accuracy, yet lifecycle state and intervention histories are built on those labels; ablations show the state matters but not that stable identities are the cause.","rationale":"The paper is a genuine attempt to make repair progress explicit, and the controlled ablations are informative: Table 3 shows that replacing the issue state with raw archive synthesis or score-based parent selection degrades performance substantially, and the expanded Tau-Bench evaluation in Appendix E.2 adds credibility. Reading in good faith, the empirical effect of the full system is plausible and the code is released. The load-bearing weakness is not the absence of an error bar per se, but the unmeasured correctness of the state that the method is named after. The issue state is the entire proposed control mechanism, and every update step (association, lifecycle transition, intervention-outcome recording) is produced by an unvalidated LLM diagnostic pipeline. The paper itself flags this in Section 6, so the concern is not manufactured. A reader cannot currently tell whether the gains come from stable issue identities and lifecycle tracking or from the structured diagnostic scaffolding around them. This is exactly the reader's weakest assumption, and I find no independent evidence in the manuscript that resolves it. Because the mechanism is coherent, the ablations are informative, and the limitation is explicitly acknowledged, the appropriate verdict remains CONDITIONAL rather than ACCEPT or REJECT. I would not change the reader's verdict; I would add the oracle-state comparison as the first required follow-up.","tokens_in":22075,"tokens_out":10057,"duration_ms":111120,"concrete_test":"Run ADIAS on the full Tau-Bench retail split (or the Table 3 benchmarks) twice, keeping all code, prompts, optimization budget, and random initialization identical. Condition A uses the standard learned Issue Manager. Condition B uses an oracle state update: issue identities, lifecycle transitions, and intervention-outcome records are taken from independent human annotation of the same diagnostic reports and trajectories instead of the model's association. Report final Pass-1/Pass-2 scores and the per-round agreement between learned and oracle issue records (for example adjusted Rand index plus merge/split counts). Pre-register a tolerance, for example conclusion unchanged if Condition A is within 2 score points of Condition B. If A matches B, unmeasured diagnosis is not load-bearing; if A drops materially, the central claim is conditioned on issue-association quality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that persistent issue identities and lifecycle states are the operative control mechanism is not yet supported, because the quality of the state itself is never measured. Section 3.3.1 defines issue association only by 'normalized failure category, affected capability, execution context, and supporting evidence'; no precision, recall, merge/split counts, or identity-stability statistics are reported. Section 6 explicitly says: 'We do not separately report the accuracy of failure labels or issue attributions.' Lifecycle transitions such as CONFIRMED-FIXED after αmin=2 absences and the intervention-outcome histories are computed from these unverified labels, so the entire persistent state could be built on noisy or drifting identities. The Archive-Wide Synthesis ablation (Table 3) shows a 40.7% drop when the state is removed, but that only demonstrates that the structured state is useful as a context/control mechanism; it does not establish that the benefit comes from stable identities, correct lifecycle status, or accurate outcome attribution. If issue associations are unstable, the observed gains could instead come from the diagnostic scaffolding, the external prior, or the extra structured context, leaving the issue-centric formulation without direct support.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes issue-centric agent optimization, a paradigm for automated design of interactive agent systems in which cross-round repair progress is stored in an explicit persistent issue state (issue identity, priority, lifecycle status, supporting evidence, and intervention-outcome history) rather than reconstructed from candidate archives. The proposed framework ADIAS combines a diagnostic agent, a one-shot external task-prior search, an Issue Manager that maintains this state, and an issue-guided optimizer that jointly selects repair targets and revision directions for focused full-code modifications. Across five interactive benchmarks (Tau-Bench, ALFWorld, TextCraft, WebShop, ScienceWorld), ADIAS reports an average score of 78.4 versus 62.6 for the strongest baseline (a 25.2% relative improvement), consistent gains across four backbone models on Tau-Bench, and ablations showing performance drops of up to 40.7% when the issue state is removed or decoupled from control. The paper also provides repeated-rollout and full-distribution checks for Tau-Bench.","tokens_in":22273,"tokens_out":6610,"duration_ms":64589,"significance":"If the claims hold, issue-centric state is a potentially valuable design principle for automated agent design, shifting the field from candidate archives to explicit repair-state representation. The paper has notable strengths: the authors ship code, use a matched backbone model for the optimizer and the task agent, run controlled ablations for external priors, round-level diagnosis, archive-wide synthesis, and parent-selection policies, and include an expanded Tau-Bench distribution check. The cross-model results and repeated-rollout analysis on Tau-Bench strengthen robustness. However, the headline improvement is not yet established with statistical confidence across all benchmarks, and the central mechanism—stable issue identities and lifecycle tracking—is not directly measured. The contribution is significant but currently conditional on additional evidence.","major_comments":[{"comment":"The headline claim of a 25.2% average improvement over the strongest baseline is computed from single-rollout scores in Table 1, and repeated-rollout evidence is provided only for Tau-Bench (Table 6). Because every other benchmark is reported without variance or significance estimates, the cross-benchmark claim is not yet statistically supported. Please add repeated rollouts or error bars for all five benchmarks, or explicitly restrict the claim to the settings with robustness evidence.","section":"Section 5.1, Table 1; Section E.1, Table 6"},{"comment":"The task subsets are filtered in ways that may be correlated with the proposed mechanism: WebShop retains only instances requiring more than 10 non-redundant interactions, ScienceWorld retains only instances whose reference trajectories complete within 30 steps and whose task types have more than five eligible cases, and the main Tau-Bench setting uses only return-related tasks. The expanded-distribution check in Appendix E.2 is limited to Tau-Bench. These filters could select long-horizon, high-variance tasks that specifically reward explicit issue-state tracking, so the five-benchmark claim should be validated on unfiltered distributions or accompanied by an analysis of how the filter affects each method.","section":"Section 4 (Benchmarks), Appendix D.1"},{"comment":"The persistent issue state is the paper's central contribution, but the quality of its core inputs is never measured. Section 6 states that failure-label and issue-attribution accuracy are not reported, and Section 3.3.1 defines issue association without any precision, recall, merge/split, or identity-stability statistics. The lifecycle transitions (e.g., CONFIRMED-FIXED after alpha_min = 2 absences) and intervention-outcome histories are built on these unverified labels, so the ablation in Table 3 cannot distinguish the hypothesis that stable issue identities drive improvement from the hypothesis that any structured summarization of the same evidence would help. Please report diagnostic and association accuracy, or run controlled manipulations such as oracle labels, shuffled identities, or no identity persistence.","section":"Section 3.3.1 (Persistent Issue State), Section 6 (Limitations)"},{"comment":"ADIAS is the only method that receives external task-level priors, and Table 3 shows that removing this prior costs up to 30 points on TextCraft (91.0 to 61.0). The main comparison in Table 1 therefore confounds the issue-centric mechanism with an information advantage shared by no baseline. To support the claim that ADIAS outperforms candidate-centric methods because of issue-centric optimization, the external prior should either be given to all baselines or be ablated in the main results, or the authors should quantify the interaction between the prior and the issue-state mechanism.","section":"Section 3.3.1, Appendix C.1, Table 3"}],"minor_comments":[{"comment":"The lifecycle parameter alpha_min is fixed at 2 with no sensitivity analysis; since the CONFIRMED-FIXED transition is part of the claimed mechanism, please report a small sweep or justify the choice.","section":"Section 3.3.1"},{"comment":"The paragraph explaining Table 6 is ambiguous about the relationship between the two individual passes and the Pass 1/Pass@2 aggregates; please clarify that a 'same score' across the two passes does not imply identical per-task outcomes, as the text briefly notes.","section":"Section E.1, Table 6"},{"comment":"The line styles for Best-Candidate Revision, Latest-Candidate Continuation, and ADIAS are hard to distinguish in small print; please use larger markers and a legend with explicit test scores.","section":"Figure 4"},{"comment":"The paper states that AHE and Meta-Harness were adapted for general interactive tasks but gives no details on the adaptation; please document what was changed so readers can assess fairness.","section":"Appendix D.2"},{"comment":"There are minor typographical issues, such as the running title 'AUTOMATEDDESIGN' missing a space; please proofread the final version.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the paper proposes a plausible and potentially useful control-state abstraction, and the empirical package is transparent about its limitations. The main risks to publication are (i) the unverified accuracy of the issue state, which is the paper's central mechanism, and (ii) the external-prior asymmetry that confounds the baseline comparison. Both are addressable with additional experiments, so I do not recommend rejection, but the authors should be required to provide the state-quality evidence and a fairer comparison before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is real and worth engaging with: instead of organizing cross-round experience around candidate agents, ADIAS maintains an explicit persistent issue state with identities, lifecycle status, evidence, and intervention-outcome histories, and uses that state to guide full-code repairs. As far as I can tell from the literature review, no prior full-code method does exactly this. The paper also does some things well beyond the idea: the ablations are thoughtfully structured to separate evidence, representation, and control; there is an expanded evaluation on the full Tau-Bench retail distribution; and they report repeated rollouts with Pass@2 for Tau-Bench, which is more than many papers in this area bother to do. The cross-model robustness check is a nice touch too.\n\nThe soft spots are the ones the reader's report and the stress-test note flag, and I think they're sized correctly. The biggest issue is that the mechanism's core—reliable issue identification and stable identity association—is never measured. Section 6 says outright that they do not report accuracy of failure labels or issue attributions. That matters because lifecycle transitions and intervention-outcome histories are computed on top of those labels. The ablations show that removing the issue state or decoupling it from control hurts a lot, but that establishes that the structured state is useful as a control signal, not that stable identities are the causal ingredient. The external-prior injection is also a fairness gap: ADIAS gets a one-shot external search over task-level knowledge that the baselines don't receive, so part of the gain could come from that prior rather than from issue-centric tracking. And the main 25.2% average is over subsets (WebShop filtered to long trajectories, ScienceWorld to short ones, Tau-Bench to returns only), with no error bars for the non-Tau-Bench results. The Tau-Bench expansion mitigates this for one benchmark, but not for the others.\n\nNone of this makes the paper a reject. The central claim is conditional, and the paper itself is honest about the biggest condition. A serious referee could push for diagnostic accuracy measurements, variance reporting on all benchmarks, and a full-distribution check on at least one non-Tau-Bench environment, but the direction is sound and the evidence assembled so far is genuinely informative. I'd send this to peer review and expect it to come back stronger after revision.\n\nWho's it for: anyone working on automated agent design, LLM-based program repair, or self-improving agents. It deserves a serious referee despite the open questions.","headline":"Genuinely new control-state idea for automated agent design, with honest ablations and a useful Tau-Bench expansion, but the central mechanism's accuracy is unmeasured and the headline gains rest on subsets and single rollouts.","tokens_in":22848,"tokens_out":1456,"would_cite":true,"duration_ms":16760,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces issue-centric agent optimization, which stores repair progress as a persistent issue state and uses it to control full-code agent revisions, and reports that this outperforms the strongest candidate-centric baseline…","keywords":["issue-centric agent optimization","automated agent design","persistent issue state","full-code revision","interactive benchmarks","agent harness optimization","lifecycle tracking","LLM agents"],"falsifier":"Run ADIAS on one benchmark with issue identities randomly permuted at every round while keeping all other components fixed: if performance stays near the reported level, the persistent issue state is not the operative cause. Alternatively, trace diagnostic reports across rounds and count how often the same underlying failure is assigned a new identity instead of being matched to an existing one; a high mismatch rate would violate the stable-identity assumption the method depends on.","tokens_in":21821,"feed_emoji":"🤖","tokens_out":7238,"duration_ms":65616,"temperature":0.7,"pith_summary":"Automated agent design normally keeps cross-round experience organized around candidate agents, so the optimizer has to reconstruct what went wrong and what has already been tried at every round. This paper argues that repair progress should instead be stored as an explicit persistent issue state, with stable identities, priorities, lifecycle status, supporting evidence, and intervention-outcome histories, and that this state should directly control which code revision comes next. The paper instantiates this as ADIAS, a full-code design method, and reports that it outperforms the strongest baseline by 25.2% on average across five interactive benchmarks and stays ahead across four backbone models. Controlled ablations report drops of up to 40.7% when the persistent state is removed or demoted to mere context. The paper's own limitation is that it does not measure the accuracy of the failure diagnosis and issue association that the state depends on.","feed_headline":"Issue-centric search beats candidate-centric agent design by 25.2%","feed_subtitle":"ADIAS tracks unresolved issues and guides full-code repairs, beating the strongest baseline by 25.2%.","key_machinery":"The machinery that carries the argument is the persistent issue state together with the Issue Manager that maintains it. Each issue record $e_t^i = (id_i, q_i^t, s_i^t, B_i^t, U_i^t)$ stores a stable identity, a priority, a lifecycle status, supporting evidence, and an intervention-outcome history, and the state is updated after every round by associating new diagnostic findings with existing identities or creating new ones. Issue-guided optimization then jointly selects target issues and a revision plan, chooses a parent agent, and applies a focused full-code patch, so the state is an operational control signal rather than background context. Supporting components, including a one-shot external prior search, an evidence-querying diagnostic agent, and a trajectory database, feed the state, but the paper's controlled ablations isolate the state itself as the load-bearing difference.","core_discovery":"The central claim is that in long-horizon interactive environments, the bottleneck in automated agent design is not the expressiveness of the search space but the continuity of the repair objective. ADIAS therefore augments the usual archive of candidates with a persistent issue state $E_t$, in which each issue carries a stable identity $id_i$, priority $q_i^t$, lifecycle status $s_i^t$ (active, tentatively fixed, confirmed fixed, or regressed), supporting evidence $B_i^t$, and intervention-outcome history $U_i^t$. The Issue Manager associates newly diagnosed failures with existing identities, records targeted interventions separately from side effects on other issues, and chooses both the parent candidate and the revision direction from this state. The paper reports a 25.2% average improvement over the strongest baseline across Tau-Bench, ALFWorld, TextCraft, WebShop, and ScienceWorld, consistent gains across four backbone models on Tau-Bench, and ablation drops of up to 40.7% when the issue state is removed or used only as context.","pith_inferences":["A natural extension is to measure diagnostic and issue-attribution accuracy explicitly; if identities drift, adding a de-duplication or identity-verification step should improve ADIAS further, a test the paper does not run.","The same persistent-issue control could be ported to prompt-level and architecture-level agent search, because those settings face the same repair-targeting bottleneck even without full-code edits.","The lifecycle state, active, tentatively fixed, confirmed fixed, or regressed, doubles as a human-auditable record of why each code change was made, which could support inspection and safety review of self-improving agents.","Because ADIAS can edit its own issue manager and code improver, its optimization loop is itself evolvable; a focused experiment would check whether diagnostic quality improves over rounds."],"forward_implications":["Automated agent design becomes a cumulative repair process: later rounds inherit the issue-specific outcome of earlier interventions instead of re-deriving repair intent from raw candidate records.","The 25.2% average gain over the strongest baseline across five interactive benchmarks implies that making repair progress explicit is more important than expanding the editable code space alone.","Consistent gains across four backbone models on Tau-Bench suggest the issue-centric control state transfers across model families, so the method is not tuned to one language model's failure patterns.","Ablation drops of up to 40.7% imply that neither external priors nor extra diagnosis explain the gains; the issue state must actively steer parent selection and revision planning.","The expanded Tau-Bench retail result, 81.7 versus 68.7 for the strongest baseline, indicates that the advantage is not confined to the cheaper return-intent subset used in the main experiments."],"supporting_citations":[{"why":"Supplies the Tau-Bench customer-service benchmark and the repeated-execution Pass@k and Pass k metrics used in robustness checks.","marker":"Yao et al., 2025"},{"why":"Supplies the ALFWorld long-horizon embodied planning benchmark, one of the five evaluation environments.","marker":"Shridhar et al., 2021"},{"why":"Supplies the TextCraft compositional crafting benchmark and the qualitative optimization traces used in the analysis.","marker":"Prasad et al., 2024"},{"why":"Supplies the WebShop web-navigation benchmark and its normalized task score.","marker":"Yao et al., 2022"},{"why":"Supplies the ScienceWorld scientific-reasoning environment benchmark.","marker":"Wang et al., 2022"},{"why":"Provides the ReAct-style handcrafted fixed baseline with task-level memory.","marker":"Yao et al., 2023"},{"why":"Provides the Meta-Harness full-code candidate-centric baseline and the mechanism reused in the Archive-Wide Synthesis ablation.","marker":"Lee et al., 2026"},{"why":"Provides DGM-H, the strongest full-code baseline that ADIAS outperforms by 25.2% on average.","marker":"Zhang et al., 2026a"},{"why":"Provides SkillOpt, the prompt-level skill-document baseline that illustrates the limits of a restricted editable scope.","marker":"Yang et al., 2026"},{"why":"Provides AHE, the architecture-level harness optimization baseline.","marker":"Lin et al., 2026a"}],"fun_headline_variants":["ADIAS tracks issues to beat candidate agents by 25.2%","Issue-centric agent design outperforms by 25.2%","Persistent issue state drives 25.2% agent design gain","Fix the issue, not the agent: ADIAS gains 25.2%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the failure-detection component reliably identifies distinct problems and that the Issue Manager attaches each newly observed failure to the correct persistent identity across rounds; the paper does not measure this accuracy, so noisy or drifting labels could let the reported gains come from external priors, extra diagnosis, or search budget instead of issue-centric tracking.","fun_headline_variants_meta":{"raw":{"variants":["ADIAS tracks issues to beat candidate agents by 25.2%","Issue-centric agent design outperforms by 25.2%","Persistent issue state drives 25.2% agent design gain","Fix the issue, not the agent: ADIAS gains 25.2%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000229,"raw_usage":{"total_tokens":1481,"prompt_tokens":950,"completion_tokens":531,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":453}},"tokens_in":566,"tokens_out":531,"duration_ms":5395,"temperature":1.0,"reasoning_tokens":453,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T04:25:37.941931+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ADIAS on one benchmark with issue identities randomly permuted at every round while keeping all other components fixed: if performance stays near the reported level, the persistent issue state is not the operative cause. Alternatively, trace diagnostic reports across rounds and count how often the same underlying failure is assigned a new identity instead of being matched to an existing one; a high mismatch rate would violate the stable-identity assumption the method depends on.","supporting_citations":[{"cited_title":"-bench: A benchmark for tool-agent-user interaction in real-world domains","cited_arxiv_id":null,"evidence_quote":"Supplies the Tau-Bench customer-service benchmark and the repeated-execution Pass@k and Pass k metrics used in robustness checks."},{"cited_title":"Alfworld: Aligning text and embodied environments for interactive learning","cited_arxiv_id":null,"evidence_quote":"Supplies the ALFWorld long-horizon embodied planning benchmark, one of the five evaluation environments."},{"cited_title":"Adapt: As-needed decomposition and planning with language models","cited_arxiv_id":null,"evidence_quote":"Supplies the TextCraft compositional crafting benchmark and the qualitative optimization traces used in the analysis."},{"cited_title":"Scienceworld: Is your agent smarter than a 5th grader? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.\\ 11279--11298, 2022","cited_arxiv_id":null,"evidence_quote":"Supplies the ScienceWorld scientific-reasoning environment benchmark."}],"review_version":1}