{"id":"73d7721c-45d7-4abe-adb8-5707071b57ec","arxiv_id":"2608.11772","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"DARC diagnoses each task family's dominant failure mode on development tasks, prunes mismatched recovery interventions, and freezes a cost-aware fallback policy, improving success while reducing environment steps or retrieval budget on ALFWorld, AppWorld, and XBRL Finance.","lead":"An AI assistant that fails at a task usually gets more instructions, which often makes things worse. This paper builds DARC, a system that first studies how agents fail on development tasks, then freezes a small set of matching repair strategies that improve task success and cut cost across three benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The only controlled isolation of the diagnosis step (Appendix H, Table 14) shows no significant held-out accuracy benefit; diagnosis's measured gains are search-space reduction and selection stability, so the central claim that pruning mismatched interventions drives task-success gains is not yet…","rationale":"The reader's verdict is CONDITIONAL and identifies the Section 4.2 short-circuit proxy as the weakest assumption. I agree that the offline evidence matrix validity is explicit and untested, but the more directly falsifying gap is the absence of a controlled diagnosis-removal comparison on AppWorld and Finance. The paper's own ALFWorld control in Table 14 shows that removing diagnosis does not significantly reduce held-out accuracy, which undercuts the causal framing that pruning mismatched interventions produces the success gains. Since the reader's conditional verdict already asks for code, per-task outcomes, and guard weights, my concern points to a specific missing experiment rather than a change in verdict: the paper should either run the full-library cascade on the other two benchmark families or visibly narrow the claim about what diagnosis alone contributes to held-out success. I therefore retain CONDITIONAL rather than moving to REJECT, because the headline deployment numbers could still be reproducible and useful even if the diagnosis-necessity story is weakened. The concern is not about the authors' integrity; it is about the match between the reported causal mechanism and the evidence supplied.","tokens_in":23229,"tokens_out":6679,"duration_ms":77042,"concrete_test":"Run the Appendix H full-library cascade control on AppWorld and XBRL Finance with the same Algorithm 1 policy-distillation protocol, the same intervention library, verifier, cost penalty, maximum chain length, selection splits, and deployment budgets as DARC, but omit the diagnosis restriction that narrows R_m. Compare held-out TGC/SGC, Finance macro accuracy, environment steps, and retrieval budget against the corresponding DARC rows in Tables 3, 4, and 9. If the undiagnosed cascade matches or exceeds DARC on AppWorld and Finance, as it does on ALFWorld, then the claim that diagnosis-guided pruning drives the headline success gains is falsified and only offline search efficiency would remain; if DARC outperforms the undiagnosed cascade on those families, the diagnosis-necessity claim would gain the controlled support it currently lacks.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is causal: DARC's diagnosis-guided restriction is what turns failures into selective correction and produces the held-out gains over base agents and generic playbooks. The paper's own strongest control, Table 14 in Appendix H, isolates the diagnosis step by running the identical validation-selected policy search with and without the diagnosis restriction on ALFWorld. The full-library cascade without diagnosis reaches 98.51% on valid_unseen versus 99.25% for the diagnosed cascade, a difference of +0.75 pp with 95% CI [0.00, 2.24], which the paper reports as not significant. The measured benefits of diagnosis are therefore a 10x smaller search space (40 vs 400 candidate policies) and greater selection stability (14 vs 34 distinct chains under resampling), not higher held-out task success. This matters because the abstract and Section 1 attribute the success improvement to pruning mismatched interventions and making recovery selective. If the same or equivalent accuracy is obtainable by a validation-selected cascade without any diagnosis step, then the headline gains on ALFWorld are not evidence for the diagnosis-guided mechanism. Appendix F explicitly states that the corresponding factorial decomposition for the AppWorld and Finance harnesses is not available, and Table 14's full-library cascade has not been run on those families. Consequently, for the two settings where DARC reports its largest average gains, there is no controlled evidence that diagnosis restriction contributes to the outcome rather than the validation-selected policy search itself. The short-circuit proxy the reader flagged is a real selection-validity risk, but the main tables are deployment results; a wrong proxy would show up in deployment, whereas the missing diagnosis control can coexist with good deployment numbers while still invalidating the stated mechanism. The diagnosis-control gap is therefore the more load-bearing missing piece for the paper's central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DARC, a diagnosis-guided recovery harness for language agents. The idea is to profile development-set failures to identify a dominant failure mode per task family, restrict the candidate recovery library to interventions that match that failure mode, and then distill a frozen, verifier-scored success-cost fallback policy over the restricted set. The authors evaluate the protocol on ALFWorld (action-validity recovery), AppWorld (procedural-knowledge recovery), and XBRL Finance (format-precision retrieval-budget recovery), reporting consistent average gains over a base agent, ICL, MIPROv2, GEPA, and ACE, together with reduced environment steps or retrieval budget. They also present a matched-information fairness analysis for ALFWorld, a factorial decomposition of the recovery harness, a validation-selected full-library cascade control in Appendix H, and a preliminary weight-space training extension. The paper is transparent about its supervision accounting, its short-circuit policy-estimation assumption, and the scope of its controlled diagnosis isolation experiment.","tokens_in":23449,"tokens_out":5349,"duration_ms":58800,"significance":"If the causal claim held, DARC would be a practical and reproducible recipe for making self-correction selective rather than context-expanding, and the three-domain instantiation would give the claim useful breadth. The paper has genuine strengths: the supervision accounting in Table 2 is explicit; the matched-information analysis in Table 7 directly addresses the strongest fairness objection to the ALFWorld result; the action-view/prompt factorial in Appendix F reports exact task counts with temperature 0; and Appendix H reports a matched full-library cascade control rather than arguing only by ablation. These are the right controls to run. However, the paper's own strongest control — the diagnosis-isolation experiment in Appendix H — shows no statistically significant accuracy benefit from removing diagnosis, and the generic/mismatched ablation in Table 6 is partially degenerate. The significance of the work therefore depends on whether the claims can be reframed as 'diagnosis improves search efficiency and selection stability without sacrificing accuracy' rather than 'diagnosis is what drives the task-success improvements.'","major_comments":[{"comment":"The only controlled isolation of the diagnosis step does not support the causal claim that pruning mismatched interventions drives held-out task-success gains. The full-library cascade without diagnosis reaches 98.51% on ALFWorld valid_unseen versus 99.25% for the diagnosed cascade, with a 95% CI of [0.00, 2.24] that the paper itself reports as not significant. The measured benefits of diagnosis are a 10x smaller search space and greater selection stability, not higher accuracy. The abstract and Section 1 attribute the headline improvements to making correction selective and pruning mismatched interventions; those statements overstate what Table 14 establishes. The claims should be revised to separate accuracy gains from search-efficiency and stability gains, or additional evidence must show that diagnosis is causally responsible for the task-success improvements.","section":"Appendix H, Table 14"},{"comment":"The diagnosis-necessity ablation is weakened by the ALFWorld row, where the Generic and Mismatched conditions both report exactly 52.24% success, so the comparison cannot distinguish a generic playbook from a mismatched intervention set on the benchmark with the largest gains. On AppWorld, the Mismatched condition (64.88% TGC) exceeds the Generic condition (61.90% TGC), contrary to the expected ordering stated in the text; the text acknowledges this only after presenting the claim that the correct policy outperforms both. The conclusion that 'correct > generic > mismatched' holds consistently is therefore not supported by the table as presented. The authors should explain the identical ALFWorld values and either provide a non-degenerate generic/mismatched contrast or soften the claim.","section":"Section 6.1, Table 6"},{"comment":"The controlled evidence for the diagnosis mechanism is limited to ALFWorld. Appendix F explicitly states that the corresponding factorial decomposition for the AppWorld and Finance harnesses is not available, and Appendix G indicates that the full-library cascade has not been extended to those families. For the two settings where DARC reports its largest average gains, there is no matched validation-selected cascade without diagnosis, so the central claim that diagnosis-guided restriction is responsible for the AppWorld and Finance improvements is unsupported. The generic and mismatched rows in Table 6 for those families do not use the same policy-search protocol as DARC, and their confidence intervals overlap for AppWorld. The authors should either run the full-library cascade on AppWorld and Finance or explicitly restrict the causal claim to ALFWorld.","section":"Section 6.2 and Appendix F"},{"comment":"The offline evidence-matrix proxy is load-bearing for the distilled policy, and the paper's own explicit assumption is not validated outside resettable environments. The short-circuit estimator in Eqs. (4)-(6) assumes later interventions are evaluated only after an earlier intervention fails and that the verifier can attribute success and cost to the invoked intervention from the same initial state. For ALFWorld this is justified by resettable evaluation, and for Finance by statelessness, but for AppWorld the deployment semantics are described only as 'controlled task attempts' with verifier-attributed success. No experiment demonstrates that the offline matrix predicts deployment outcomes for AppWorld, where intervention attempts may interact with state. The assumption is disclosed, which is good, but a validation check — e.g., comparing offline-selected policy predictions against actual frozen-policy deployment on a held-out set — is needed before the AppWorld distilled policy can be taken as a reliable proxy.","section":"Section 4.2, Table 1"}],"minor_comments":[{"comment":"The 'Answer Labels' column header is confusing; since it repeats the distinction defined in Table 2, the table should either rename the column to 'Ground-truth answer labels used in adaptation' or add a footnote cross-referencing Table 2.","section":"Table 4"},{"comment":"The scenario-cluster bootstrap confidence intervals are reported only for DeepSeek-V4-Flash; the Qwen3.5-27B and Qwen3.6-27B results in Table 3 do not have the same uncertainty quantification, so the statistical-significance claim is not uniform across the three backbones.","section":"Section 5.2, Figure 2"},{"comment":"The note that the diagnosed row re-selects a cascade over the four action-validity harnesses, and therefore the 99.25% figure is not directly comparable to the fixed single-harness DARC row of 90.30% in Table 3, is easy to miss; this difference in selection protocol should be stated prominently in the main text when Table 14 is discussed.","section":"Table 14"},{"comment":"The sentence in Section 6.4 says the distilled policy 'stays in the top macro-accuracy band' while reducing the mean budget to 1.5; Table 9 shows that fixed k=2 and k=4 also achieve 94.25%, so the policy-selection advantage over k=2 is only 0.25 pp on macro accuracy, which should be acknowledged in the surrounding text.","section":"Section 6.4, Table 9"},{"comment":"For ALFWorld, the target-tuned row is reported as '–', so the claim that the frozen policy 'preserves much of the target-tuned performance' is not substantiated on that benchmark; the sentence should either report the target-tuned ALFWorld number or restrict the generalization statement to Finance.","section":"Section 5.3, Table 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is transparent and the primary harness result is large and credible on ALFWorld, but the central causal narrative is stronger than the controlled evidence. The most important fix is reframing or additional experimentation around Appendix H: as written, the diagnosis-isolation experiment shows diagnosis buys efficiency and stability, not accuracy. If the authors reframe the contribution accordingly and add the missing full-library cascades for AppWorld and Finance, or explicitly limit the causal claim to ALFWorld, the paper could be acceptable. I would not reject on circularity grounds, since the evaluation is external and the policy is selected on training-set verifier feedback without test labels."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it gives agent builders a concrete, reusable recipe: profile development-set failures, prune mismatched recovery interventions, and freeze a verifier-selected success-cost policy. That packaging is new, and the headline gains on ALFWorld, AppWorld, and XBRL Finance are real enough to take seriously. Second, the paper's own strongest control shows that the diagnosis step itself buys no significant held-out accuracy. That matters because the abstract and intro attribute the gains to pruning mismatched interventions.\n\nWhat is genuinely good: the supervision accounting (Table 2) is honest, the matched-information analysis on ALFWorld (Table 7) directly addresses the privileged-information concern, and the factorial decomposition (Table 13) is a clean piece of evidence that the gain is an interaction between the restricted action view and the recovery prompt, not either alone. The full-library cascade in Table 14 is exactly the right control and the authors deserve credit for running it. That is more rigor than most agent papers.\n\nThe soft spots, in proportion. The red flag is Table 6: ALFWorld Generic and Mismatched both come in at 52.24%, which is precisely the ACE macro average from Table 3. That looks like a copy-paste or data-entry error, and it needs an erratum before I'd trust the ablation. More substantively, Table 14 shows diagnosis adds no significant accuracy over the validation-selected full-library cascade (98.51% vs 99.25%, CI [0.00, 2.24]). The measured benefits of diagnosis are a 10x smaller search space and more stable selection, not higher task success. So the central causal claim is not supported by the paper's own control. For AppWorld and Finance, the corresponding factorial decomposition is unavailable, so we don't know whether the interaction structure generalizes outside ALFWorld. Also missing: code or data, despite a claim of released per-task outcomes; the Eq. 8 guard coefficients are undisclosed; and most numbers lack error bars, with only the AppWorld bootstrap reported.\n\nNone of that kills the paper. The main tables do support the average-improvement claim, and the protocol is a plausible design for a real problem. It deserves serious peer review, but I'd want the Table 6 red flag resolved, code/data released, and the causal language narrowed to what Table 14 actually shows: diagnosis improves search efficiency and stability while matching accuracy. For anyone building recovery interfaces for LLM agents, this is a useful read with clear caveats.","headline":"A genuinely useful recipe for turning agent failures into selective recovery, but the paper's own controls undercut the causal claim that diagnosis drives the gains.","tokens_in":24173,"tokens_out":2042,"would_cite":false,"duration_ms":21656,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A diagnosis-guided recovery harness lifts agent success and cuts recovery cost.","keywords":["language agents","self-correction","failure diagnosis","recovery policy","action validity","retrieval budget","agent evaluation","prompt optimization"],"falsifier":"Run DARC on a benchmark where interventions cannot be evaluated from the same initial state — for example, tasks in which attempting the first intervention irreversibly consumes a shared budget or mutates state before the second intervention starts. If the offline evidence matrix no longer predicts deployment, the distilled short-circuit policy's success should fall toward the generic-playbook level or below the base agent. A complementary test is constructing a task family whose failures split evenly between two distinct modes: the frozen single-mode diagnosis should underperform a full-library cascade on that family.","tokens_in":22932,"feed_emoji":"🔧","tokens_out":6965,"duration_ms":62990,"temperature":0.7,"pith_summary":"This paper argues that broad language-agent self-correction fails not because agents lack corrective context but because they lack diagnosis: a coarse task failure mixes incompatible recovery signals, and generic playbooks add irrelevant material that interferes while inflating cost. The authors propose DARC, a harness that profiles development-set failures to identify a task family's dominant failure mode, prunes mismatched interventions from a shared recovery library, and distills a short, verifier-selected success-cost policy that is frozen before deployment. The claim is that this causal order — decide what kind of failure can be repaired, then decide how much recovery evidence to spend — improves average task success over base agents and broad playbooks while reducing environment steps or retrieval budget across ALFWorld, AppWorld, and XBRL Finance. A sympathetic reader would care because the paper offers a practical route to more reliable agents in domains where compiler-like feedback is absent: making failures actionable before making contexts larger.","feed_headline":"Diagnosis-first repair lifts agent success from 40% to 90%","feed_subtitle":"A frozen recovery harness prunes mismatched interventions, beating generic playbooks on ALFWorld, AppWorld, and Finance.","key_machinery":"The central object is the recovery harness $H_m=(m,\\mathcal{R}_m,\\pi_m)$, built in two stages. Failure diagnosis profiles development-set failures and prunes the candidate intervention library; policy distillation enumerates bounded intervention chains, scores each by the objective $J(\\pi)=\\mathrm{dsucc}(\\pi)-\\lambda\\max\\{0,\\mathrm{dcost}(\\pi)-\\tau_{\\mathrm{free}}\\}$, and freezes the winner for deployment. The short-circuit rule invokes a later intervention only after earlier ones fail, with success and cost attributed by the task verifier; the coverage function over interventions is monotone and submodular, and a uniform-convergence bound over the finite chain class explains why pruning the library shrinks search risk.","core_discovery":"The central discovery is a recovery-harness construction: a frozen tuple $(m, \\mathcal{R}_m, \\pi_m)$ in which $m$ is a task-family failure diagnosis, $\\mathcal{R}_m$ is the admissible intervention subset, and $\\pi_m$ is an ordered short-circuit policy selected by training-set verifier feedback to maximize success minus a cost penalty. The paper claims this harness converts ambiguous task failure into typed recovery signals — an action-validity guard for ALFWorld, a procedural fallback chain for AppWorld, a retrieval-budget policy for Finance — and that in each setting it improves average performance (for example, ALFWorld valid_unseen success rises from 39.55% for the base agent and 54.48% for ACE to 90.30%) while reducing environment steps or retrieval budget. It further claims the improvement comes from the diagnosis-restriction pairing itself rather than from privileged information: a matched-information factorial shows the restricted action view and the recovery prompt are each near-inert in isolation and only together produce the large effect, an interaction worth +46.27 percentage points on ALFWorld.","pith_inferences":["The one-dominant-failure-mode-per-family assumption is likely to break in benchmarks whose episodes mix two failure modes; the natural extension is instance-level routing over multiple diagnoses, and the paper's own frozen design would probably lose to adaptive routing in such settings.","The matched-information factorial suggests the mechanism is interface compatibility — the restriction and the prompt only pay off together — so the harness design may transfer to other agent interfaces, such as tool-use or device-control, where the same superadditive interaction could be tested directly.","Since policies are selected on training-set verifier feedback, DARC inherits the verifier's quality: under a noisy or partial verifier, or in environments that are not resettable, the offline evidence matrix may systematically mis-estimate deployment cost, and the reported cost savings could shrink.","A direct test of the framework's boundary would be a task family where failure modes are balanced rather than dominant; the paper's diagnosis stage predicts a large degradation for the mismatched policy there, whereas a full-library cascade should stay competitive."],"forward_implications":["On tasks whose failures expose a dominant, offline-profiled failure mode, a frozen diagnosis-guided harness should match or exceed full-library recovery cascades while searching a much smaller policy space; the paper reports this on ALFWorld (99.25% versus 98.51% test accuracy, not significantly different, over a 10x smaller search space).","Because the policy is frozen and verifier-selected, it transfers across related splits and tasks when the failure mode is similar: Finance cross-task transfer stays within about 0.5 to 3 percentage points of target-tuned performance, and an ALFWorld action-validity policy distilled on one split reaches 90.30% on the unseen split.","Cost-aware ordering means expensive recovery evidence is spent only after cheaper interventions fail: the Finance policy retrieves a mean of 1.5 demonstrations at top macro accuracy, and the ALFWorld harness solves more episodes at every step budget while cutting average environment steps by more than half.","The success term is a monotone submodular coverage function, so adding interventions beyond a short policy yields rapidly diminishing recoverable success at growing cost; saturated policy chains are therefore unlikely to add value.","The main results degrade gracefully but not uniformly: on AppWorld challenge splits, the ACE baseline remains competitive or better on some metrics, and the paper interprets the gains as average-utility evidence rather than universal dominance."],"supporting_citations":[{"why":"Supplies the ALFWorld embodied task family and the resettable evaluation protocol used to validate the action-validity harness.","marker":"[65]"},{"why":"Supplies the AppWorld multi-application procedural tasks and the controlled task-attempt protocol for the procedural-recovery fallback.","marker":"[73]"},{"why":"Supplies the XBRL Finance extraction tasks whose exact-match evaluation defines the format-precision failure mode.","marker":"[45]"},{"why":"The broad recovery-playbook baseline (ACE) that DARC must beat on all three benchmarks.","marker":"[94]"},{"why":"Instruction and demonstration optimization baseline (MIPROv2) used in the main comparisons.","marker":"[48]"},{"why":"Reflective prompt-evolution baseline (GEPA) used in the main comparisons.","marker":"[3]"},{"why":"Establishes the compiler-like execution-feedback paradigm that DARC extends to tasks without such feedback.","marker":"[13]"},{"why":"Supports the motivation that irrelevant context can distract LLM reasoning, justifying pruning of recovery material.","marker":"[63]"},{"why":"Supports the motivation that models fail to use long contexts robustly, justifying restricted interventions over playbooks.","marker":"[42]"}],"fun_headline_variants":["DARC harness lifts agent success from 40% to 90%","Selective self-correction: DARC prunes recovery to boost agents","Diagnosis-first recovery cuts steps, lifts ALFWorld to 90%","Frozen recovery harness turns failures into typed signals","DARC: less context, more success via diagnosis-guided repair"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes each recovery intervention can be evaluated independently from the same initial task state, with the verifier attributing success and cost to the invoked intervention, so that the offline evidence matrix is a valid proxy for deployment behavior.","fun_headline_variants_meta":{"raw":{"variants":["DARC harness lifts agent success from 40% to 90%","Selective self-correction: DARC prunes recovery to boost agents","Diagnosis-first recovery cuts steps, lifts ALFWorld to 90%","Frozen recovery harness turns failures into typed signals","DARC: less context, more success via diagnosis-guided repair"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000559,"raw_usage":{"total_tokens":2713,"prompt_tokens":1060,"completion_tokens":1653,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":676,"completion_tokens_details":{"reasoning_tokens":1563}},"tokens_in":676,"tokens_out":1653,"duration_ms":11599,"temperature":1.0,"reasoning_tokens":1563,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:29:04.374155+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DARC on a benchmark where interventions cannot be evaluated from the same initial state — for example, tasks in which attempting the first intervention irreversibly consumes a shared budget or mutates state before the second intervention starts. If the offline evidence matrix no longer predicts deployment, the distilled short-circuit policy's success should fall toward the generic-playbook level or below the base agent. A complementary test is constructing a task family whose failures split evenly between two distinct modes: the frozen single-mode diagnosis should underperform a full-library cascade on that family.","supporting_citations":[{"cited_title":"Appworld: A controllable world of apps and people for benchmarking interactive coding agents","cited_arxiv_id":null,"evidence_quote":"Supplies the AppWorld multi-application procedural tasks and the controlled task-attempt protocol for the procedural-recovery fallback."},{"cited_title":"Finer: Finan- cial numeric entity recognition for xbrl tagging","cited_arxiv_id":null,"evidence_quote":"Supplies the XBRL Finance extraction tasks whose exact-match evaluation defines the format-precision failure mode."},{"cited_title":"Agenticcon- text engineering: Evolving contexts for self-improving language models","cited_arxiv_id":null,"evidence_quote":"The broad recovery-playbook baseline (ACE) that DARC must beat on all three benchmarks."},{"cited_title":"Optimizing instructions and demon- strations for multi-stage language model programs","cited_arxiv_id":null,"evidence_quote":"Instruction and demonstration optimization baseline (MIPROv2) used in the main comparisons."},{"cited_title":"Chi, Nathanael Schärli, andDennyZhou","cited_arxiv_id":null,"evidence_quote":"Supports the motivation that irrelevant context can distract LLM reasoning, justifying pruning of recovery material."},{"cited_title":"Liu, Kevin Lin, John Hewitt, Ashwin Paran- jape, Michele Bevilacqua, Fabio Petroni, and Percy Liang","cited_arxiv_id":null,"evidence_quote":"Supports the motivation that models fail to use long contexts robustly, justifying restricted interventions over playbooks."}],"review_version":1}