Pith. sign in

REVIEW 4 major objections 5 minor 38 references

Agent systems improve faster when their recurring failure modes are induced from their own traces and reused as a shared feedback vocabulary, rather than being re-diagnosed in free text on every run.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 21:09 UTC pith:5II3QEFS

load-bearing objection Real, reusable failure-taxonomy method with honest search/runtime evidence; the faithfulness validation might be in-sample and some headlines are single-run, but it deserves refereeing. the 4 major comments →

arxiv 2607.16387 v2 pith:5II3QEFS submitted 2026-07-17 cs.SE cs.AI

Fantastic Adaptive Taxonomies and How to Use Them

Mert Cemri , Andrei Cojocaru , Melissa Pan , Shu Liu , Shubham Agarwal , Alexander Krentsel , Jay Tang , Kannan Ramchandran
show 4 more authors
Joseph E. Gonzalez Matei Zaharia Alex Dimakis Ion Stoica
This is my paper
classification cs.SE cs.AI
keywords adaptive failure taxonomyLLM agentsfailure modesexecution tracesagent-system searchruntime monitoringtrajectory selectioninter-annotator agreement
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Most procedures that improve an LLM agent without retraining—trajectory selection, prompt and workflow optimization, runtime monitoring—read execution traces for feedback, but raw traces are long and instance-specific and free-text critiques are discarded with each run. The paper argues that an agent system should instead maintain an explicit failure taxonomy induced from its own traces: named, evidence-grounded failure codes that are validated for consistent applicability before deployment and reused wherever failure feedback is needed. The central discovery is that this single artifact improves all three trace-consuming procedures it was tested on—search, runtime monitoring, and trajectory selection—while being compact, human-faithful, and adaptive to the target system. If the paper is right, the cheapest improvement to an agent system may be to first learn, in the system's own terms, how it fails.

Core claim

The paper introduces AdaMAST, a pipeline that converts a target system's execution traces into a validated failure taxonomy: named failure codes organized along three fixed axes—system-level, role-specific, and domain-specific—with every code name, definition, and evidence pattern induced from the traces. No code is hand-authored and no trace is human-annotated; a taxonomy is accepted only after independent annotators apply it consistently to held-out traces, and it is refined online as the system evolves. Because the codes are grounded in observed evidence and partitioned by intervention point, the same taxonomy serves as a shared feedback interface for three otherwise dissimilar consumers:

What carries the argument

The central object is the induced failure taxonomy: a compact set of named failure codes arranged along three fixed axes (system-level, role-specific, domain-specific), where each code's name, definition, role label, and evidence pattern are generated from the target system's own traces rather than authored in advance. The three axes are fixed so taxonomies remain comparable across systems and every code maps to an intervention point, while the codes inside are entirely adaptive; a taxonomy is certified for deployment only when an inter-annotator agreement gate shows independent annotators applying it consistently to held-out traces, and online refinement merges, adds, or relabels codes as t

Load-bearing premise

The inter-annotator agreement gate and the human-faithfulness check are both run with LLM annotators rather than human experts, so if LLMs share systematic biases in labeling failures, the certification that a taxonomy is consistently applicable and human-faithful could be vacuous.

What would settle it

Have independent human experts apply an induced taxonomy to the same held-out traces used for the agreement gate; if human pairwise agreement falls well below the LLM panel's κ≥0.75 threshold, the certification collapses as an artifact of shared LLM bias.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Any trace-consuming improvement procedure—search, monitoring, selection—can consume the same induced taxonomy, so the one-time cost of building it is amortized across every downstream run.
  • Taxonomy-coded diagnoses in mutation prompts outperform free-form reflection on all five tested benchmarks, with gains of +3.5 to +7.5 percentage points under matched search budgets.
  • Anchoring runtime checkpoints to the induced vocabulary raises SWE-agent resolution on SWE-bench Verified Mini from 60% with free-text reflection and 68% with a fixed checklist to 70%, and Claude Code from 64.0% to 70.7%.
  • A verifier built on the induced codes improves best-of-5 accuracy on Terminal-Bench 2.0 by 8–15 points over no selection, and held-out cross-validation meets or exceeds the same-pool result on non-saturated harnesses.
  • The induced vocabulary is compact (~18× compression, 89% unique trace signatures), more faithful to expert failure labels than a hand-crafted vocabulary, and adaptive across domains (mean pairwise code overlap 0.14).

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The amortization argument implies the taxonomy's value grows with the number of consumers and runs sharing it; the strongest test would be a long-horizon deployment where one vocabulary simultaneously feeds search, monitoring, and selection.
  • Because passing trajectories fire fewer codes than failing ones, a simple 'fewest fired codes' heuristic could serve as a cheap, LLM-free selection baseline when a learned verifier is too expensive.
  • Role-specific codes appear only when the architecture has differentiated roles, so the evolving share of role codes could serve as an implicit measure of architectural complexity and a trigger for splitting a flat agent into roles.
  • The low cross-domain code overlap but higher overlap on system-level failures suggests a testable warm-start: induce a new domain's taxonomy seeded with the system-level axis of an existing one, then measure whether agreement and downstream gains arrive sooner.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes AdaMAST, an LLM-driven pipeline that induces a compact failure taxonomy from an agent system's execution traces along three fixed axes (system-level, role-specific, domain-specific), gates it by inter-annotator agreement among four LLM annotators, and then consumes the taxonomy as feedback in three settings: evolutionary agent-system search, runtime self-monitoring, and trajectory selection. The authors report consistent search gains over free-form reflection on five benchmarks, runtime resolution improvements on SWE-bench Verified Mini under two harnesses, best-of-5 selection gains on Terminal-Bench 2.0, and artifact-level properties (≈18× compression, area-κ=0.682 vs. 0.516 against expert TRAIL labels, mean cross-domain Jaccard 0.14). The paper is unusually thorough in its appendices, including matched comparisons, multi-seed ablations on small benchmarks, wrong-domain transfer checks, cost accounting, and explicit caveats about single-run transfer studies and non-causal mechanism traces.

Significance. If the central claims hold, AdaMAST provides a reusable, evidence-grounded failure vocabulary that can serve as an amortized feedback interface for multiple trace-consuming procedures, which is a genuinely useful idea. The paper's strengths include: a clear separation of induction from deployment, matched experimental comparisons that hold the surrounding procedure fixed, honest acknowledgment of format-null and substrate-sensitivity results, a functional-substitution protocol with two consumer models, and an unusually detailed appendix set with prompts, hyperparameters, and code-release information. The main weaknesses are that the 'human-faithful' validation rests on an unstated and potentially circular TRAIL split and on LLM annotators rather than human annotators, and that several headline numbers come from single-seed or single-run evaluations. These issues are fixable with additional experiments or more measured claims, so the paper is promising but not yet conclusive.

major comments (4)
  1. [§5.1, Appendix F.1] The TRAIL faithfulness comparison may be in-sample. The paper states 'No TRAIL labels are used to induce the AdaMAST taxonomy,' but it never states whether the 117 TRAIL traces used for evaluation were excluded from the trace pool used to induce the taxonomy. Since AdaMAST induces code names and evidence patterns directly from traces, evaluating on the same traces would make the κ=0.682 vs. 0.516 gap an expected consequence of fitting the evaluation set, not evidence of better correspondence to expert judgment. Please state the split explicitly; if the same traces were used, re-run with a held-out trace split or leave-traces-out cross-validation.
  2. [§3.1, §5.1 (Table 22)] The 'human-faithful' claim is supported only by LLM annotators. The IAA gate in §3.1 certifies LLM-LLM consistency, not correctness, and §5.1 measures agreement between an LLM panel and expert gold, not between human annotators and the induced taxonomy. If LLMs share systematic labeling biases, both the κ≥0.75 gate and the reported κ advantage could be vacuous. The same-model vs. cross-family comparison is a useful control, but it does not establish human applicability. Please either add a human-annotation study on a held-out subset or downgrade the claim from 'human-faithful' to 'LLM-applied alignment with expert gold.'
  3. [§4.3, Table 3] The runtime headline rests on a single seed for the SWE-agent arm: 35/50 vs. 34/50 is a one-instance difference, so the claimed +2 pp margin over MAST is within binomial noise. The Base→Reflexion→MAST→AdaMAST ladder is suggestive, but the SWE-agent arm needs multiple seeds or confidence intervals before the ordering can be treated as evidence. At minimum, the paper should not present the +2% over MAST as a robust finding. The Claude Code arm, with three seeds, is stronger.
  4. [§4.4, Appendix D] The trajectory-selection result does not isolate the adaptive vocabulary. AdaMAST-Judge routes codes through a learned forward selector and heuristic features; Tables 16 and 18 show that wrong-domain taxonomies and a 5-code truncation achieve the same accuracy, and D.5 indicates the selector carries most of the marginal effect. The paper concedes this in the text, but the abstract and contributions credit the selection gain to AdaMAST-Judge without this qualification. Please either add an ablation that varies only the vocabulary under the same selected feature set or explicitly reframe the selection section as an integration study rather than evidence for the induced vocabulary.
minor comments (5)
  1. [§1] There is a stray '0' after 'AdaMAST(Adaptive Multi-Agent System Failure Taxonomies), 0 which induces'. Please remove.
  2. [§3] 'we avoidinferdeliberately' is missing a space/formatting; should read 'we avoid infer deliberately.'
  3. [Table 5] Header contains a typo: 'V ocabulary' should be 'Vocabulary.'
  4. [§5.4 / Appendix E] The compression numbers are reported with several proxies (gzip, raw characters, token-cost proxy). Please clarify which number corresponds to the '∼18×' headline and state whether the taxonomy text is amortized in each reported figure.
  5. [Appendix B.4] The sentence 'No TRAIL labels are used to induce the AdaMAST taxonomy' is easy to misread as 'no TRAIL data are used.' Please rephrase to explicitly state whether TRAIL traces were used and how the evaluation split was handled.

Circularity Check

1 steps flagged

TRAIL faithfulness comparison may be in-sample: no held-out trace split is stated

specific steps
  1. fitted input called prediction [Section 5.1, Table 5; Appendix F.1]
    "A four-LLM panel applies theAdaMAST-induced TRAIL taxonomy using span-grounded prompts, one round of peer deliberation, and majority vote. ... We use TRAIL only as an external validation set; no TRAIL labels are used to induce theAdaMASTtaxonomy."

    The paper's only external check that the induced codes are human-recognizable compares κ=0.682 (AdaMAST-induced TRAIL taxonomy) with κ=0.516 (TRAIL hand-crafted) on the same 117 TRAIL traces. The stated exclusion is labels only: 'no TRAIL labels are used to induce the AdaMAST taxonomy.' The manuscript never states that the 117 TRAIL traces were excluded from induction, and it explicitly calls the taxonomy 'AdaMAST-induced TRAIL.' If the taxonomy was induced from those traces, the vocabulary is built from the same traces whose expert labels are the prediction target; the κ advantage is then an in-sample fit, not evidence that the codes generalize to human judgment. The absence of a stated held-out trace split is the gap.

full rationale

Most of the paper's downstream evaluations are external and controlled: search uses held-out OlympiadBench sets and multi-seed small-benchmark replications, runtime uses a cross-domain OfficeQA transfer with a taxonomy induced from separate baseline transcripts, trajectory selection uses leave-one-task-out cross-validation, and functional substitution predicts held-out run success from a taxonomy induced from a disjoint pool. The taxonomy is not fitted to the target outcome (resolution/accuracy), and the LLM-based IAA gate is explicitly labeled by the authors as certifying consistency, not correctness ('This gate certifies that the taxonomy is consistently applicable; it does not certify that its codes are correct'). Self-citations to MAST and AdaEvolve are present but used as baselines/backbones, not as load-bearing justification for the central claims. The one significant circularity concern is the TRAIL faithfulness validation: the paper says TRAIL labels were not used to induce the taxonomy, but never states that the 117 TRAIL traces themselves were held out. Since the taxonomy is called 'AdaMAST-induced TRAIL,' the κ=0.682 vs 0.516 comparison may be in-sample, making the 'human-faithful' claim partly circular. This is a single, conditional gap in an otherwise externally validated paper, so the score is moderate rather than high.

Axiom & Free-Parameter Ledger

10 free parameters · 5 axioms · 0 invented entities

The paper's central claim depends on the chosen axes, the LLM-based validation protocol, and the representativeness of trace samples. These are domain assumptions rather than standard math. No new physical or formal entities are postulated.

free parameters (10)
  • IAA kappa threshold = 0.75
    Chosen by hand as the deployment gate for inter-annotator agreement; not fit to the target result.
  • coverage floor = 0.70
    Chosen by hand for the IAA gate.
  • warmup rounds (Nwarm) = 5
    Hyperparameter in search experiments; balances trace pool quality vs. compute.
  • stagnation window (w) = 10 iterations
    Hyperparameter for online refinement trigger.
  • stagnation threshold (epsilon) = 0.10
    Hyperparameter for refinement trigger.
  • min refine interval (rmin) = 10 iterations
    Prevents back-to-back refinement.
  • population size = 20
    Standard GA hyperparameter.
  • number of islands = 3
    Standard GA hyperparameter.
  • migration interval = 8 iterations
    Standard GA hyperparameter.
  • feedback codes injected (K) = 5-7
    Top codes by frequency injected into mutation prompt.
axioms (5)
  • domain assumption The three fixed axes (system-level, role-specific, domain-specific) partition failures by intervention point.
    Section 3 states the axes are a 'practical organizational scaffold', not uniquely optimal; the whole taxonomy structure depends on this partition.
  • domain assumption LLM annotators' inter-annotator agreement (kappa>=0.75) is a sufficient gate for taxonomy deployability.
    Section 3.1: the IAA gate uses four LLM annotators; no human check is performed before deployment.
  • domain assumption The induced taxonomy generalizes from the induction pool to held-out traces.
    Section 3.1: the pipeline induces from a sample and assumes the sample represents the system's failure modes.
  • standard math Cohen's kappa and Jaccard similarity are appropriate measures for taxonomy agreement and overlap.
    Used throughout Section 3.1 and Section 5 without discussion.
  • domain assumption The target system's traces are representative of its failure modes.
    The pipeline's input is a pool of traces; if the pool is not representative, the taxonomy will not capture recurring failures.

pith-pipeline@v1.3.0-alltime-deepseek · 33095 in / 13551 out tokens · 109087 ms · 2026-08-01T21:09:16.169611+00:00 · methodology

0 comments
read the original abstract

An agent system's execution traces record how it fails, and procedures that improve such a system without changing model weights (trajectory selection, prompt and workflow optimization, runtime monitoring) read these traces for feedback. Yet raw traces are a poor medium for accumulating that feedback: long, instance-specific, and lacking a stable vocabulary for recurring failures. We argue that an agent system should instead maintain an explicit representation of how it fails, induced from its own behavior and reusable wherever failure feedback is needed. AdaMAST builds this representation by converting a target system's traces into a compact, evidence-grounded failure taxonomy: named failure codes organized along three fixed axes (system-level, role-specific, and domain-specific), with every name, definition, and evidence pattern induced from the traces; no code is hand-authored, no trace human-annotated. The taxonomy is not merely a post-hoc diagnostic but a shared feedback interface, improving agents in three ways. In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form reflection on all five benchmarks we test. At runtime, taxonomy feedback raises SWE-agent's resolution on SWE-bench Verified Mini from 60% with free-text reflection to 70%, and improves Claude Code from 64.0% to 70.7% as a runtime skill. In trajectory selection, AdaMAST-Judge, a verifier built on the induced codes, improves best-of-5 accuracy on Terminal-Bench 2.0 by 8-15 points over Pass@1. The vocabulary itself is compact (an order-of-magnitude compression that preserves trace distinctions), human-faithful (matching expert failure annotations more closely than a hand-crafted reference vocabulary), and adaptive (taxonomies induced for different domains share few codes). Adaptive failure taxonomies close the loop between the traces agents produce and the procedures that improve them.

Figures

Figures reproduced from arXiv: 2607.16387 by Alexander Krentsel, Alex Dimakis, Andrei Cojocaru, Ion Stoica, Jay Tang, Joseph E. Gonzalez, Kannan Ramchandran, Matei Zaharia, Melissa Pan, Mert Cemri, Shubham Agarwal, Shu Liu.

Figure 1
Figure 1. Figure 1: Three adaptive failure axes. The axes of intervention are fixed; the concrete code names, definitions, role labels, and evidence patterns are induced from the target system’s traces. Shown: representative codes from the OlympiadBench evolution run, with verbatim evidence excerpts from the run’s judge outputs in quoted italics (Appendix G.2); unquoted lines under C-codes are induced code descriptions ( [PI… view at source ↗
Figure 2
Figure 2. Figure 2: AdaMAST pipeline. ANALYSIS, CURATION, and CONSOLIDATION produce a draft taxon￾omy through an eight-step induction pipeline (Appendix B.1). INTER-ANNOTATOR AGREEMENT gates deployment: four independent LLM annotators label stratified held-out traces under a five-phase deliberation protocol, over up to five rounds of five traces each; the taxonomy is accepted once mean pairwise area-level agreement reaches κ … view at source ↗
Figure 3
Figure 3. Figure 3: Search gains across benchmarks. Post-search accuracy of AdaMAST and the LLM Reflection baseline in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: OlympiadBench search mechanism. (a) Best dev score per iteration for the vanilla baseline and AdaMAST; both share the AdaEvolve substrate and seed architecture, and only the mutation feedback differs. The open circle marks the vanilla run’s exhaustion at iteration 26 (dev score 0.30). (b) Held-out accuracy on 655 problems across the seed, vanilla, MAST-guided, and AdaMAST-guided architectures. We read this… view at source ↗
Figure 5
Figure 5. Figure 5: Fired codes and trajectory outcome. The ForgeCode AdaMAST taxonomy is applied to every trial of the 17 ForgeCode swing tasks. (a) Distribution of fired codes for passing versus failing trajectories; triangles mark the medians (3 for passing, 4 for failing). (b) Expected swing-task accuracy of a fewest-fired-codes selector versus uniform selection (58% → 67%, +9 pp). trial with the fewest fired codes reache… view at source ↗
Figure 6
Figure 6. Figure 6: Top induced failure codes for TheoremQA (13 codes: 4 system, 9 domain). Bars show [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Top induced failure codes for Frontier-CS (25 codes: 5 system, 13 role, 7 domain). Bars [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Top induced failure codes for SWE-bench under the SWE-agent harness (30 codes: 8 [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Cross-domain code overlap. Pairwise Jaccard between the induced taxonomies for six evolution domains. (a) Full code sets (mean off-diagonal 0.14). (b) Each domain projected onto a universal failure backbone: a fixed set of cross-cutting failure tokens such as context exhaustion, looping, and format violation (mean 0.50) [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Long-tail structure of induced taxonomies. (a) Lorenz curves: cumulative share of code firings against the rank-sorted fraction of codes; the top ∼20% of codes account for the bulk of firings on every run. (b) Same data on a log-rank axis. The shape is consistent across OlympiadBench, Frontier-CS, and three TheoremQA seeds, indicating that the concentration is a property of the induced vocabulary rather t… view at source ↗
Figure 11
Figure 11. Figure 11: is the raw best-dev-score-per-iteration curve for the OlympiadBench AdaMAST and vanilla runs, with the two annotated score-jumps at iter 69 (verifier gate) and iter 89 (planner upstream). The vanilla run saturates at iter 26 with max dev score 0.30; AdaMAST crosses to 0.40 at iter 10, 0.50 at iter 69, and 0.55 at iter 89. 0 20 40 60 80 100 Evolution iteration 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Best dev score (20… view at source ↗
Figure 12
Figure 12. Figure 12: extends this analysis to per-code lift on score-jump iterations across three benchmarks. For each benchmark we score every code by its firing rate on the score-jump iterations (j) and on the remaining iterations (o), then plot the top few by lift (j − o). The bench-level top codes are the role￾and domain-specific ones identified in the verbatim judge outputs above, not generic A.1/A.5 codes; this is the c… view at source ↗
Figure 13
Figure 13. Figure 13: Failure-axis distribution shifts during evolution. Share of judge-fired codes by axis in three windows of the OlympiadBench AdaMAST run (early: iterations 1–30; mid: 31–60; late: 61–97). The B-axis share grows over the run as the architecture acquires more roles; the C-axis share contracts as the solver acquires domain-specific guard rails. 0 20 40 60 80 Evolution iteration A.7 A.8 A.5 A.4 A.1 A.2 A.3 A.6… view at source ↗
Figure 14
Figure 14. Figure 14: Code lifecycle on the OlympiadBench AdaMAST run. Each bar shows the iteration range over which a code fired at least once; the dot marks the last-firing iteration. Of the codes here, seven retire fully before iteration 80; the rest persist through the iteration-92 final taxonomy snapshot [PITH_FULL_IMAGE:figures/full_fig_p031_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Failure-mode retirement and burden decay over evolution. (a) Cumulative count of retired failure codes (codes that never fire again after a given iteration); 12 of the 28 codes have stopped firing by iteration 92, with the two score breakthroughs (dashed lines, iterations 69 and 89) marked. (b) Severity-weighted failure burden: the aggregate burden falls by 23% (16.9 to 13.0) from iteration 1 to iteration… view at source ↗
Figure 16
Figure 16. Figure 16: Vanilla vs AdaMAST per-iteration failure surface. (a) Distinct codes fired per iteration. (b) Stacked share of A/B/C axes for the vanilla run measured post-hoc against the final AdaMAST taxonomy (26 iters). (c) Same stacked share for the AdaMAST run (97 iters). Adaptive feedback does not produce more labels per iteration; it reshapes which axes the failures concentrate on. G.6 Architecture diffs The AdaMA… view at source ↗
Figure 17
Figure 17. Figure 17: Cross-arm failure profile on OlympiadBench. The post-hoc judge assigns the primary failure-mode label to each (arm, instance) cell on the 655-problem held-out set using the iteration-92 taxonomy. The vanilla best architecture (test 87.94%) is an adversarial-debate pipeline: a primary solver, two critics, an improver consolidating critiques into a revision, two more critics, a second improver, and a synthe… view at source ↗
Figure 18
Figure 18. Figure 18: Primary failure code per (arm, instance) cell under the SWE-agent harness. Two codes dominate Base’s failures: C.9 unclean patch contents (13 firings) and C.1 ad-hoc validation only (12 firings). AdaMAST in-prompt retires C.9 entirely (13 to 0) while C.1 persists at a similar level (14 vs. Base’s 12): the induced vocabulary eliminates the patch-hygiene failure but shallow validation remains the dominant r… view at source ↗
Figure 19
Figure 19. Figure 19: SWE-bench failure-area composition per arm. Each bar is the 9 instances unresolved by every arm under that condition, decomposed into three failure areas. The composition shifts upstream as the feedback surface gets richer: Verification dominates under Base/Reflexion → Implementation dominates under MAST and AdaMAST arms. of this appendix, the per-arm counts are 30/50 (Base), 34/50 (MAST runtime skill), a… view at source ↗
Figure 20
Figure 20. Figure 20: Per-arm failure code firings under the Claude Code harness. Single-seed run on 50 Verified Mini instances, three arms (Base, MAST runtime skill, and AdaMAST runtime skill). (a) Per-code firings by arm; within each bar, the solid segment counts firings on unresolved sessions and the light segment firings on resolved sessions. (b) Mass vs. predictive power: per-code total firings across the 150 sessions aga… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 13 linked inside Pith

  1. [1]

    L. A. Agrawal, S. Tan, D. Soylu, N. Ziems, R. Khare, K. Opsahl-Ong, A. Singhvi, H. Shandilya, M. J. Ryan, M. Jiang, et al. Gepa: Reflective prompt evolution can outperform reinforcement learning.arXiv preprint arXiv:2507.19457, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    Claude code

    Anthropic. Claude code. https://docs.anthropic.com/en/docs/claude-code/ overview, 2025. Accessed: 2026-05-07

    2025

  3. [3]

    Cemri, M

    M. Cemri, M. Z. Pan, S. Yang, L. A. Agrawal, B. Chopra, R. Tiwari, K. Keutzer, A. Parameswaran, D. Klein, K. Ramchandran, M. Zaharia, J. E. Gonzalez, and I. Stoica. Why do multi-agent llm systems fail? InAdvances in Neural Information Processing Systems, 2025

    2025

  4. [4]

    Cemri, S

    M. Cemri, S. Agrawal, A. Gupta, S. Liu, A. Cheng, Q. Mang, A. Naren, L. E. Erdogan, K. Sen, M. Zaharia, et al. Adaevolve: Adaptive llm driven zeroth-order optimization.arXiv preprint arXiv:2602.20133, 2026

    arXiv 2026

  5. [5]

    W. Chen, M. Yin, M. Ku, P. Lu, Y . Wan, X. Ma, J. Xu, X. Wang, and T. Xia. Theoremqa: A theorem-driven question answering dataset. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7889–7901, 2023

    2023

  6. [6]

    Cheng, A

    C.-A. Cheng, A. Nie, and A. Swaminathan. Trace is the next AutoDiff: Generative optimization with rich feedback, execution traces, and LLMs. InAdvances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  7. [7]

    Deshpande, V

    D. Deshpande, V . Gangal, H. Mehta, J. Krishnan, A. Kannappan, and R. Qian. TRAIL: Trace reasoning and agentic issue localization.arXiv preprint arXiv:2505.08638, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    D. Dua, Y . Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. InProceedings of NAACL-HLT, pages 2368–2378, 2019

    2019

  9. [9]

    Eyuboglu, M

    S. Eyuboglu, M. Varma, K. Saab, J.-B. Delbrouck, C. Lee-Messer, J. Dunnmon, J. Zou, and C. Ré. Domino: Discovering systematic errors with cross-modal embeddings. InInternational Conference on Learning Representations (ICLR), 2022. 13

    2022

  10. [10]

    Fernando, D

    C. Fernando, D. Banarse, H. Michalewski, S. Osindero, and T. Rocktäschel. Promptbreeder: Self-referential self-improvement via prompt evolution.arXiv preprint arXiv:2309.16797, 2023

    Pith/arXiv arXiv 2023

  11. [11]

    Q. Guo, R. Wang, J. Guo, et al. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  12. [12]

    S. Hu, C. Lu, and J. Clune. Automated design of agentic systems. InInternational Conference on Learning Representations (ICLR), 2025

    2025

  13. [13]

    A. S. Jia, D. Huang, N. Vytla, S. W. W. Yoo, N. Choudhury, S. Sen, J. C. Mitchell, and A. Datta. What is your agent’s GPA? a framework for evaluating agent goal-plan-action alignment.arXiv preprint arXiv:2510.08847, 2025

    arXiv 2025

  14. [14]

    Khattab et al

    O. Khattab et al. DSPy: Compiling declarative language model calls into state-of-the-art pipelines. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  15. [15]

    J. Kwok, S. Li, P. Atreya, Y . Liu, Y . Jiang, C. Finn, M. Pavone, I. Stoica, and A. Mirhoseini. Llm-as-a-verifier: A general-purpose verification framework, 2026

    2026

  16. [16]

    Lightman, V

    H. Lightman, V . Kosaraju, Y . Burda, et al. Let’s verify step by step. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  17. [17]

    S. Liu, S. Agarwal, M. Maheswaran, M. Cemri, Z. Li, Q. Mang, A. Naren, E. Boneh, A. Cheng, M. Z. Pan, et al. Evox: Meta-evolution for automated discovery.arXiv preprint arXiv:2602.23413, 2026

    arXiv 2026

  18. [18]

    Madaan et al

    A. Madaan et al. Self-refine: Iterative refinement with self-feedback. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  19. [19]

    Q. Mang, W. Chai, Z. Li, H. Mao, S. Zhou, A. Du, H. Li, S. Liu, E. Chen, Y . Wang, et al. Frontier-CS: Evolving challenges for evolving intelligence, 2025. URL https://arxiv.org/ abs/2512.15699

    arXiv 2025

  20. [20]

    McAleese, R

    N. McAleese, R. M. Pokorny, J. F. C. Uribe, E. Nitishinskaya, M. Trebacz, and J. Leike. Llm critics help catch LLM bugs.arXiv preprint arXiv:2407.00215, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y . Shin, T. Walshe, E. K. Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces.arXiv preprint arXiv:2601.11868, 2026

    Pith/arXiv arXiv 2026

  22. [22]

    Novikov, M

    A. Novikov, M. Balog, S. Chaudhuri, et al. AlphaEvolve: A coding agent for scientific and algorithmic discovery.arXiv preprint arXiv:2506.13131, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    M. Z. Pan, N. Arabzadeh, R. Cogo, et al. Measuring agents in production.arXiv preprint arXiv:2512.04123, 2026

    Pith/arXiv arXiv 2026

  24. [24]

    M. T. Ribeiro, T. Wu, C. Guestrin, and S. Singh. Beyond accuracy: Behavioral testing of nlp models with CheckList. InAssociation for Computational Linguistics (ACL), 2020

    2020

  25. [25]

    Romera-Paredes, M

    B. Romera-Paredes, M. Barekatain, A. Novikov, et al. Mathematical discoveries from program search with large language models.Nature, 625:468–475, 2024

    2024

  26. [26]

    Shinn, F

    N. Shinn, F. Cassano, A. Gopinath, et al. Reflexion: Language agents with verbal reinforcement learning. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  27. [27]

    Snell, J

    C. Snell, J. Lee, K. Xu, and A. Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    ForgeCode: An open-source multi-agent coding harness

    TailcallHQ. ForgeCode: An open-source multi-agent coding harness. https://github.com/ tailcallhq/forgecode; https://forgecode.dev, 2026. Top-ranked open-source agent on Terminal-Bench 2.0 (Pass@1 81.8% with the Forge harness). 14

    2026

  29. [29]

    Y . Wang, X. Ma, G. Zhang, Y . Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. InAdvances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2024

    2024

  30. [30]

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press. Swe- agent: Agent-computer interfaces enable automated software engineering.arXiv preprint arXiv:2405.15793, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    Yuan et al

    S. Yuan et al. EvoAgent: Towards automatic multi-agent generation via evolutionary algorithms. arXiv preprint arXiv:2406.14228, 2024

    Pith/arXiv arXiv 2024

  32. [32]

    differentiation

    M. Yuksekgonul, F. Bianchi, J. Boen, et al. TextGrad: Automatic “differentiation” via text. arXiv preprint arXiv:2406.07496, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    Zhang, J

    J. Zhang, J. Xiang, Z. Yu, et al. AFlow: Automating agentic workflow generation. InInterna- tional Conference on Learning Representations (ICLR), 2025

    2025

  34. [34]

    Zhang, A

    L. Zhang, A. Hosseini, H. Bansal, M. Kazemi, A. Kumar, and R. Agarwal. Generative verifiers: Reward modeling as next-token prediction.arXiv preprint arXiv:2408.15240, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Zhang, M

    S. Zhang, M. Yin, J. Zhang, J. Liu, Z. Han, J. Zhang, B. Li, C. Wang, H. Wang, Y . Chen, and Q. Wu. Which agent causes task failures and when? on automated failure attribution of LLM multi-agent systems.arXiv preprint arXiv:2505.00212, 2025

    Pith/arXiv arXiv 2025

  36. [36]

    Zheng, W.-L

    L. Zheng, W.-L. Chiang, Y . Sheng, et al. Judging LLM-as-a-judge with MT-Bench and chatbot arena. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  37. [37]

    Zhuge et al

    M. Zhuge et al. GPTSwarm: Language agents as optimizable graphs. InProceedings of the International Conference on Machine Learning (ICML), 2024. 15 A Generated Taxonomy Examples This appendix presents representative induced taxonomies. For each benchmark, we report a short context paragraph and the top-five failure codes by firing frequency. The full mach...

    arXiv 2024

  38. [38]

    Output: FINAL ANSWER: Please provide the problem statement

    that must agree on a category. CIs are 95% percentile intervals from 2000 trace-level bootstrap resamples. Methodology thr=1 95% CI thr=2 95% CI thr=3 95% CI Cross-family 4-panel (no grounding) 0.682[0.608,0.754]0.500[0.476,0.611]0.307[0.236,0.380] +1-round deliberation 0.692[0.620,0.764]0.590[0.517,0.665]0.439[0.354,0.524] Same-model 4×GPT-5.4 0.625[0.54...

    2000