{"id":"b38450dc-1f2d-478b-9b20-edbe90993723","arxiv_id":"2607.16387","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":8.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":10,"one_line_summary":"AdaMAST induces a compact, evidence-grounded failure taxonomy from an agent's own traces and shows it improves agent-system search, runtime reflection, and trajectory selection over free-text and fixed-taxonomy baselines.","lead":"This paper proposes AdaMAST, a system that automatically builds a compact, named failure taxonomy from an AI agent's own execution traces, then uses that taxonomy to improve the agent in search, runtime monitoring, and trajectory selection. If its results hold, agent-improvement pipelines could replace disposable free-text critiques with reusable, system-specific failure vocabularies.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TRAIL faithfulness comparison may be in-sample: no held-out split is stated, so the κ=0.682 vs 0.516 gap could reflect fitting the evaluation traces.","rationale":"The reader's weakest assumption was that LLM annotators, not human annotators, certify the taxonomy; that is a real limitation. But the sharper, more load-bearing problem is that the TRAIL faithfulness comparison may be in-sample: no held-out split is stated, so the induced vocabulary's κ advantage over the hand-crafted reference could be an artifact of having been induced from the same traces used for evaluation. This concern is concrete, checkable from the released artifacts, and directly undermines the human-faithful property in the abstract and Section 5.1. It does not invalidate the downstream search/runtime/selection results, which are objectively measured and largely independent of the faithfulness claim, so the reader's CONDITIONAL verdict remains appropriate. I set agreement to 'partial' because the reader's formal weakest_assumption was the LLM-annotator issue, while my focus is the unstated TRAIL split that the reader mentioned only in passing in the rationale.","tokens_in":33358,"tokens_out":9701,"duration_ms":89953,"concrete_test":"Inspect the released run artifacts/code to determine the TRAIL induction pool. If the taxonomy was induced on all 117 TRAIL traces, rerun the vocabulary comparison with an explicit split: induce the taxonomy on a training subset (e.g., ~60 traces) and evaluate on the held-out ~57 traces, holding the four-LLM panel and prompts fixed. If held-out area-κ for the induced vocabulary falls to ≤0.516 (the hand-crafted baseline), the faithfulness claim is an in-sample artifact and Section 5.1's vocabulary isolation must be re-interpreted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central artifact-level claim that the induced vocabulary is 'human-faithful' rests on Section 5.1: an AdaMAST-induced TRAIL taxonomy reaches area-κ=0.682 against four-expert gold, versus 0.516 for TRAIL's hand-crafted vocabulary, under a fixed LLM panel. The paper states that no TRAIL labels were used to induce the taxonomy, but it never states whether the same 117 TRAIL traces were used for induction. If they were, the comparison is in-sample: a taxonomy induced from a trace set is expected to align with the failure patterns of those traces, so the κ advantage may be an artifact of fitting the evaluation set rather than evidence of better correspondence to expert judgment. This is load-bearing because Section 5.1 is the only external validation of the codes' human-recognizable correspondence; the IAA gate itself only certifies LLM-LLM consistency, not correctness. The LLM-annotator composition compounds the issue: even with a proper split, the result demonstrates that LLM annotators using the induced codes agree with expert gold, not that human annotators can apply them. Appendix F.1 says 'No TRAIL labels are used to induce the AdaMAST taxonomy' but is silent on whether TRAIL traces were used, leaving the central faithfulness comparison with an unstated and potentially circular split.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AdaMAST, an LLM-driven pipeline that induces a compact failure taxonomy from an agent system's execution traces along three fixed axes (system-level, role-specific, domain-specific), gates it by inter-annotator agreement among four LLM annotators, and then consumes the taxonomy as feedback in three settings: evolutionary agent-system search, runtime self-monitoring, and trajectory selection. The authors report consistent search gains over free-form reflection on five benchmarks, runtime resolution improvements on SWE-bench Verified Mini under two harnesses, best-of-5 selection gains on Terminal-Bench 2.0, and artifact-level properties (≈18× compression, area-κ=0.682 vs. 0.516 against expert TRAIL labels, mean cross-domain Jaccard 0.14). The paper is unusually thorough in its appendices, including matched comparisons, multi-seed ablations on small benchmarks, wrong-domain transfer checks, cost accounting, and explicit caveats about single-run transfer studies and non-causal mechanism traces.","tokens_in":33705,"tokens_out":6188,"duration_ms":53366,"significance":"If the central claims hold, AdaMAST provides a reusable, evidence-grounded failure vocabulary that can serve as an amortized feedback interface for multiple trace-consuming procedures, which is a genuinely useful idea. The paper's strengths include: a clear separation of induction from deployment, matched experimental comparisons that hold the surrounding procedure fixed, honest acknowledgment of format-null and substrate-sensitivity results, a functional-substitution protocol with two consumer models, and an unusually detailed appendix set with prompts, hyperparameters, and code-release information. The main weaknesses are that the 'human-faithful' validation rests on an unstated and potentially circular TRAIL split and on LLM annotators rather than human annotators, and that several headline numbers come from single-seed or single-run evaluations. These issues are fixable with additional experiments or more measured claims, so the paper is promising but not yet conclusive.","major_comments":[{"comment":"The TRAIL faithfulness comparison may be in-sample. The paper states 'No TRAIL labels are used to induce the AdaMAST taxonomy,' but it never states whether the 117 TRAIL traces used for evaluation were excluded from the trace pool used to induce the taxonomy. Since AdaMAST induces code names and evidence patterns directly from traces, evaluating on the same traces would make the κ=0.682 vs. 0.516 gap an expected consequence of fitting the evaluation set, not evidence of better correspondence to expert judgment. Please state the split explicitly; if the same traces were used, re-run with a held-out trace split or leave-traces-out cross-validation.","section":"§5.1, Appendix F.1"},{"comment":"The 'human-faithful' claim is supported only by LLM annotators. The IAA gate in §3.1 certifies LLM-LLM consistency, not correctness, and §5.1 measures agreement between an LLM panel and expert gold, not between human annotators and the induced taxonomy. If LLMs share systematic labeling biases, both the κ≥0.75 gate and the reported κ advantage could be vacuous. The same-model vs. cross-family comparison is a useful control, but it does not establish human applicability. Please either add a human-annotation study on a held-out subset or downgrade the claim from 'human-faithful' to 'LLM-applied alignment with expert gold.'","section":"§3.1, §5.1 (Table 22)"},{"comment":"The runtime headline rests on a single seed for the SWE-agent arm: 35/50 vs. 34/50 is a one-instance difference, so the claimed +2 pp margin over MAST is within binomial noise. The Base→Reflexion→MAST→AdaMAST ladder is suggestive, but the SWE-agent arm needs multiple seeds or confidence intervals before the ordering can be treated as evidence. At minimum, the paper should not present the +2% over MAST as a robust finding. The Claude Code arm, with three seeds, is stronger.","section":"§4.3, Table 3"},{"comment":"The trajectory-selection result does not isolate the adaptive vocabulary. AdaMAST-Judge routes codes through a learned forward selector and heuristic features; Tables 16 and 18 show that wrong-domain taxonomies and a 5-code truncation achieve the same accuracy, and D.5 indicates the selector carries most of the marginal effect. The paper concedes this in the text, but the abstract and contributions credit the selection gain to AdaMAST-Judge without this qualification. Please either add an ablation that varies only the vocabulary under the same selected feature set or explicitly reframe the selection section as an integration study rather than evidence for the induced vocabulary.","section":"§4.4, Appendix D"}],"minor_comments":[{"comment":"There is a stray '0' after 'AdaMAST(Adaptive Multi-Agent System Failure Taxonomies), 0 which induces'. Please remove.","section":"§1"},{"comment":"'we avoidinferdeliberately' is missing a space/formatting; should read 'we avoid infer deliberately.'","section":"§3"},{"comment":"Header contains a typo: 'V ocabulary' should be 'Vocabulary.'","section":"Table 5"},{"comment":"The compression numbers are reported with several proxies (gzip, raw characters, token-cost proxy). Please clarify which number corresponds to the '∼18×' headline and state whether the taxonomy text is amortized in each reported figure.","section":"§5.4 / Appendix E"},{"comment":"The sentence 'No TRAIL labels are used to induce the AdaMAST taxonomy' is easy to misread as 'no TRAIL data are used.' Please rephrase to explicitly state whether TRAIL traces were used and how the evaluation split was handled.","section":"Appendix B.4"}],"recommendation":"major_revision","confidential_remarks":"The paper cites several same-group arXiv preprints (e.g., AdaEvolve, EvoX) as core baselines. The editor may wish to confirm that these references are publicly available and that the corresponding code/baselines can be reproduced. The TRAIL split issue and the single-seed SWE-agent result should be resolved before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — quick read on AdaMAST. The core idea is the real deal: instead of per-trace free-text critique or a fixed failure catalogue, induce a named, evidence-grounded failure taxonomy from the target system's own traces, validate it once, and reuse it across search, runtime monitoring, and trajectory selection. The three-consumer demonstration is a genuine departure from MAST and TRAIL, and the paper deserves credit for being explicit about what is and isn't causal — the OlympiadBench mechanism trace is labelled as such, and the OfficeQA transfer study is called single-run, non-confirmatory.\n\nThe strongest results are in search. Taxonomy-coded diagnoses beat free-form reflection on all five benchmarks, with multi-seed replications on TheoremQA and DROP. The compression and adaptivity numbers are also concrete: 18x compression, 89% unique trace signatures, cross-domain Jaccard 0.14. Those are clean artifact-level claims.\n\nNow the soft spots, in proportion.\n\nFirst, the 'human-faithful' claim. Section 5.1 reports area-kappa 0.682 vs 0.516 for TRAIL's hand-crafted vocabulary under a matched LLM panel. But the paper never states whether the 117 TRAIL traces were used to induce the taxonomy being tested. It only says no TRAIL labels were used. If the same traces were used for induction, the comparison is partly in-sample: a vocabulary built from a set of traces will fit the failure patterns in those traces more than a generic 20-category scheme. This matters because this section is the only external validation that the codes are human-recognizable — the IAA gate certifies only LLM-LLM consistency, and the paper admits it does not certify correctness. The annotators are LLMs, not humans, so even with a clean split you've shown LLM annotators agree with expert gold, not that people can apply the codes. This is fixable but currently load-bearing.\n\nSecond, several headline numbers are thin. Frontier-CS and OlympiadBench are single runs; the SWE-agent runtime result is one seed of 50 instances. MMLU-Pro is a two-seed mean. That's not disqualifying for a systems paper, but it makes the +10 to +20 pp runtime claims look less robust than they are.\n\nThird, the trajectory-selection gains are partly a property of the learned selector, not the vocabulary. The paper says this itself, and the wrong-domain ablation shows the selector can do just as well with mismatched taxonomies. That's honest but it means the vocabulary's contribution in selection is not isolated.\n\nWho should read this: anyone building agent-improvement loops — search over prompts/workflows, runtime self-reflection, best-of-N selection. The paper deserves a serious referee. A good reviewer should ask for the TRAIL split, a small human annotation sample, and multi-seed runs for the headline runtime results. The central argument holds up.","headline":"Real, reusable failure-taxonomy method with honest search/runtime evidence; the faithfulness validation might be in-sample and some headlines are single-run, but it deserves refereeing.","tokens_in":34219,"tokens_out":3335,"would_cite":true,"duration_ms":28522,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Agent systems improve faster when their recurring failure modes are induced from their own traces and reused as a shared feedback vocabulary, rather than being re-diagnosed in free text on every run.","keywords":["adaptive failure taxonomy","LLM agents","failure modes","execution traces","agent-system search","runtime monitoring","trajectory selection","inter-annotator agreement"],"falsifier":"Have independent human experts apply an induced taxonomy to the same held-out traces used for the agreement gate; if human pairwise agreement falls well below the LLM panel's κ≥0.75 threshold, the certification collapses as an artifact of shared LLM bias.","tokens_in":33253,"feed_emoji":"🐞","tokens_out":7898,"duration_ms":55318,"temperature":0.7,"pith_summary":"Most procedures that improve an LLM agent without retraining—trajectory selection, prompt and workflow optimization, runtime monitoring—read execution traces for feedback, but raw traces are long and instance-specific and free-text critiques are discarded with each run. The paper argues that an agent system should instead maintain an explicit failure taxonomy induced from its own traces: named, evidence-grounded failure codes that are validated for consistent applicability before deployment and reused wherever failure feedback is needed. The central discovery is that this single artifact improves all three trace-consuming procedures it was tested on—search, runtime monitoring, and trajectory selection—while being compact, human-faithful, and adaptive to the target system. If the paper is right, the cheapest improvement to an agent system may be to first learn, in the system's own terms, how it fails.","feed_headline":"Failure taxonomies induced from traces lift agents on five benchmarks","feed_subtitle":"One compact vocabulary, built from a system's own traces, improves search, runtime monitoring, and best-of-5 selection—at ~18× compression.","key_machinery":"The central object is the induced failure taxonomy: a compact set of named failure codes arranged along three fixed axes (system-level, role-specific, domain-specific), where each code's name, definition, role label, and evidence pattern are generated from the target system's own traces rather than authored in advance. The three axes are fixed so taxonomies remain comparable across systems and every code maps to an intervention point, while the codes inside are entirely adaptive; a taxonomy is certified for deployment only when an inter-annotator agreement gate shows independent annotators applying it consistently to held-out traces, and online refinement merges, adds, or relabels codes as t","core_discovery":"The paper introduces AdaMAST, a pipeline that converts a target system's execution traces into a validated failure taxonomy: named failure codes organized along three fixed axes—system-level, role-specific, and domain-specific—with every code name, definition, and evidence pattern induced from the traces. No code is hand-authored and no trace is human-annotated; a taxonomy is accepted only after independent annotators apply it consistently to held-out traces, and it is refined online as the system evolves. Because the codes are grounded in observed evidence and partitioned by intervention point, the same taxonomy serves as a shared feedback interface for three otherwise dissimilar consumers:","pith_inferences":["The amortization argument implies the taxonomy's value grows with the number of consumers and runs sharing it; the strongest test would be a long-horizon deployment where one vocabulary simultaneously feeds search, monitoring, and selection.","Because passing trajectories fire fewer codes than failing ones, a simple 'fewest fired codes' heuristic could serve as a cheap, LLM-free selection baseline when a learned verifier is too expensive.","Role-specific codes appear only when the architecture has differentiated roles, so the evolving share of role codes could serve as an implicit measure of architectural complexity and a trigger for splitting a flat agent into roles.","The low cross-domain code overlap but higher overlap on system-level failures suggests a testable warm-start: induce a new domain's taxonomy seeded with the system-level axis of an existing one, then measure whether agreement and downstream gains arrive sooner."],"forward_implications":["Any trace-consuming improvement procedure—search, monitoring, selection—can consume the same induced taxonomy, so the one-time cost of building it is amortized across every downstream run.","Taxonomy-coded diagnoses in mutation prompts outperform free-form reflection on all five tested benchmarks, with gains of +3.5 to +7.5 percentage points under matched search budgets.","Anchoring runtime checkpoints to the induced vocabulary raises SWE-agent resolution on SWE-bench Verified Mini from 60% with free-text reflection and 68% with a fixed checklist to 70%, and Claude Code from 64.0% to 70.7%.","A verifier built on the induced codes improves best-of-5 accuracy on Terminal-Bench 2.0 by 8–15 points over no selection, and held-out cross-validation meets or exceeds the same-pool result on non-saturated harnesses.","The induced vocabulary is compact (~18× compression, 89% unique trace signatures), more faithful to expert failure labels than a hand-crafted vocabulary, and adaptive across domains (mean pairwise code overlap 0.14)."],"fun_headline_variants":["AdaMAST: traces become failure taxonomies that lift agents","Built from traces, AdaMAST taxonomy boosts search, runtime, selection","Automatic failure taxonomy from traces improves agents on 5 benchmarks","Agent traces transform into failure codes: AdaMAST lifts agents 8-15 pts"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The inter-annotator agreement gate and the human-faithfulness check are both run with LLM annotators rather than human experts, so if LLMs share systematic biases in labeling failures, the certification that a taxonomy is consistently applicable and human-faithful could be vacuous.","fun_headline_variants_meta":{"raw":{"variants":["AdaMAST: traces become failure taxonomies that lift agents","Built from traces, AdaMAST taxonomy boosts search, runtime, selection","Automatic failure taxonomy from traces improves agents on 5 benchmarks","Agent traces transform into failure codes: AdaMAST lifts agents 8-15 pts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000556,"raw_usage":{"total_tokens":2547,"prompt_tokens":869,"completion_tokens":1678,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":1608}},"tokens_in":613,"tokens_out":1678,"duration_ms":10235,"temperature":1.0,"reasoning_tokens":1608,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T21:09:16.169611+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent human experts apply an induced taxonomy to the same held-out traces used for the agreement gate; if human pairwise agreement falls well below the LLM panel's κ≥0.75 threshold, the certification collapses as an artifact of shared LLM bias.","supporting_citations":[],"review_version":1}