{"id":"39552298-7d26-4965-8e5b-ba3361bec89e","arxiv_id":"2508.13143","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Across 204 runs of three agent frameworks on 34 tasks, only about half are successful, and failures fall into planning, execution, and response-generation categories.","lead":"This paper benchmarks three open-source LLM agent frameworks on 34 coding tasks and reports an overall success rate near 50%. The authors then group the observed failures into a three-tier taxonomy covering planning, execution, and response generation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run evaluation with no seeds, temperatures, or confidence intervals leaves the 50% success-rate claim and the GPT-4o-mini outperformance narrative within sampling noise; a rerun with multiple seeds is needed before the quantitative baseline can be accepted.","rationale":"The reader's weakest_assumption correctly flags benchmark curation and annotation reliability as threats to external validity and to the taxonomy. I agree those are real, but the most directly load-bearing issue for the quantitative half of the central claim is more basic and is visible from the paper itself: the experiments are one shot per cell. Even if the 34 tasks are accepted as representative, a binomial proportion with n=34 has a wide confidence interval (roughly 33–66% for 50%), and claims about specific framework/category strengths rest on n=12 or n=4 per cell. This is not a matter of external artifacts: the missing information is seeds, temperatures, and replicate runs, and the decisive test is a rerun. If reruns shift the 50% rate or invert the mini-versus-4o comparison, then the failure taxonomy, which is built from the logs of the first run, also becomes less trustworthy because the observed failure set would not be the typical one. This does not mean the authors' analysis is wrong; it means the current evidence is insufficient to separate systematic findings from sampling noise. I therefore retain the reader's CONDITIONAL verdict and propose multi-seed replication with confidence intervals as the check that would settle the concern.","tokens_in":7728,"tokens_out":12815,"duration_ms":145540,"concrete_test":"Rerun all 34 tasks under the three frameworks and two GPT-4o backbones with at least 5 independent seeds at fixed temperature settings (or at temperature 0 and 0.2 if defaults were used), and recompute Tables II and III with exact binomial 95% confidence intervals. Also compare the failure logs across seeds. If any per-framework overall success rate moves by more than 10 percentage points, or the GPT-4o-mini-over-GPT-4o advantage appears in fewer than 4 of 5 seeds, the quantitative baseline and the overthinking explanation in Sec. IV-A should be treated as unconfirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The main quantitative claims in Secs. III–IV are computed from one execution per (task, framework, backbone) cell. Section IV-B1 reports 34 tasks × 3 frameworks × 2 LLMs = 204 runs, and no replicate runs, random seeds, or sampling temperatures are stated. LLM-based tool-execution pipelines are stochastic, so Tables II and III are point estimates from a single Bernoulli trial per cell. With only n=34 per overall row, and n=12 or n=4 per category, the reported differences are within plausible sampling noise: the '100%' file-operation cells are 4/4 successes, and a single different seed would move them by 25 points. The observed GPT-4o-mini advantage (TaskWeaver 58.82 vs 50.00; AutoGen 50 vs 38.24) and the 'overthinking' attribution to GPT-4o in Sec. IV-A cannot be distinguished from randomness on this evidence. Moreover, the 104 failures that seed the taxonomy are one realization of the same stochastic process; rerunning can change which tasks fail and therefore which causes are 'most frequent' in Fig. 3. Without exact binomial confidence intervals or multi-seed replication, the paper's headline 'approximately 50%' and its cross-framework/cross-model narratives are not yet robust.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a benchmark of 34 programmable tasks in three categories (web crawling, data analysis, file operations), evaluates three open-source agent frameworks (TaskWeaver, MetaGPT, AutoGen) with two LLM backbones (GPT-4o and GPT-4o-mini), and reports an overall task completion rate of about 50% across 204 runs. The authors manually inspect the 104 logged failures and construct a three-tier failure taxonomy (task planning, task execution, response generation) with 19 specific causes, illustrate three common failure cases, and propose two design recommendations: learning-from-feedback planning and early-stop/navigation mechanisms. The paper's central contributions are empirical: a baseline success rate, a failure taxonomy, and associated mitigation advice.","tokens_in":7981,"tokens_out":5490,"duration_ms":53315,"significance":"If the headline success rate and the taxonomy are robust, the paper would provide a useful quantitative reference and a structured vocabulary for diagnosing failures in LLM-based agents. Strengths include the executable-task benchmark with human-verified ground truth, pinned framework and model versions, the release of data and evaluation code, and a manual log analysis that grounds the taxonomy in concrete execution traces rather than speculation. However, the statistical support for the headline numbers and, especially, for cross-framework and cross-model comparisons is currently insufficient, and the manual annotation lacks an inter-annotator agreement measure. These issues need to be addressed before the empirical claims can be taken as baseline results.","major_comments":[{"comment":"All quantitative results are point estimates from one execution per (task, framework, backbone) cell, with no random seeds, temperatures, or replication stated. With only 34 tasks total and per-category cell sizes of 12, 18, and 4 (e.g., a 100% file-operation score is 4/4), the observed differences are within sampling noise. For instance, the claimed GPT-4o-mini advantage for TaskWeaver (58.82% vs 50.00%) corresponds to 20/34 vs 17/34, which is not significant under an exact binomial comparison, and the AutoGen comparison (50.00% vs 38.24%) is similarly fragile. The paper should report exact binomial confidence intervals or, preferably, run multiple seeds per cell and report means and variances; every comparative claim in Section IV-A should be qualified accordingly.","section":"Section IV-A, Tables II and III"},{"comment":"The claim that GPT-4o underperforms because of 'overthinking' (a conflict between task-planning processes and built-in safety constraints such as denying web scraping) is based on qualitative inspection of selected logs rather than a systematic count or a controlled comparison. Since the observed performance difference is within sampling noise, this explanation is not currently supported by the evidence presented. Please either quantify the frequency of this pattern across all GPT-4o failures or explicitly frame it as an anecdotal observation.","section":"Section IV-A, 'overthinking' paragraph"},{"comment":"The failure taxonomy is the paper's second main contribution, but its reliability is not established. The annotations were produced by the three authors with no reported inter-annotator agreement statistic and no independent validation set. Because the 104 failure logs are a single stochastic realization and the leaf-level counts in the taxonomy (e.g., the 'most frequent' cause highlighted in Figure 3) inherit that variability, the paper should at minimum report agreement statistics such as Cohen's kappa, and preferably also assess the sensitivity of the taxonomy to the particular run.","section":"Section IV-B1"},{"comment":"The task-selection criteria (executable, automatically evaluable, and at least partially solvable by an agent) are transparent, but they mean the benchmark is a curated subset rather than a representative sample of 'programmable tasks.' The 50% figure should therefore be consistently described as benchmark-specific, and the authors should report the size of the initial task pool and how many tasks were excluded at each screening step so that readers can judge selection bias.","section":"Section III-A"}],"minor_comments":[{"comment":"Please report the maximum iteration limit, sampling temperature, and any random seed or deterministic settings; these details are needed to interpret Figure 2 and the tables.","section":"Section III-C"},{"comment":"The text says there are three planning causes but the figure lists four planning-level leaves, including 'Infinite loop with same response'; align the numbering with the figure or explain the relationship between 'failed self-refinement' and 'infinite loop.'","section":"Section IV-B2 and Figure 3"},{"comment":"The sentence 'Both TaskWeaver and AutoGen achieve perfect scores in File Operations' refers to the GPT-4o-mini results in Table III, but as written it appears to describe both tables; clarify which model is being discussed.","section":"Section IV-A"},{"comment":"The sentence 'The superior performance of GPT-4o-mini is consistent with prior research findings [28]' cites a relevance-assessment study that is not directly about agent task completion; either cite evidence from agent benchmarks or temper the claim.","section":"Section IV-A"},{"comment":"The x-axis definition ('maximum iteration numbers') and the aggregation across tasks and models should be stated in the caption or text, and no error bars or replication are shown.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for an empirical AI/systems venue, and I see no fundamental circularity in the main measurement: the success rates are compared against human-created ground truth and the taxonomy is a qualitative labeling of the logs. The primary barrier is statistical rigor: the single-run design undermines the cross-model and cross-framework claims, while the absent inter-annotator agreement weakens the taxonomy's reliability. These are fixable with additional experiments and reporting changes, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a small but honest empirical study of LLM agent failures. It contributes a 34-task benchmark and a three-tier failure taxonomy, and it reports that three open-source frameworks paired with two LLM backbones complete about half of the tasks. The headline number is plausible and consistent with prior work, but the quantitative precision is weaker than the paper implies.\n\nThe task design is genuinely executable and automatically evaluated, with human-verified ground truth. The taxonomy (planning/execution/response generation, 19 causes) is a useful consolidation of known failure modes, and the three case studies in Figure 4 are concrete and instructive. The authors also point to a public repo and pin a commit hash for TaskWeaver, which is the right instinct for reproducibility.\n\nThe main soft spot is statistical. Every (task, framework, backbone) cell was run once. With n=34 overall and per-category n as small as 4, a single seed change moves a 100% score to 75%. No confidence intervals or significance tests are reported, so the claims that GPT-4o-mini outperforms GPT-4o and that TaskWeaver leads on structured tasks are within sampling noise. The failure taxonomy is also a single manual labeling of one realization of the stochastic process; no inter-annotator reliability is reported, and rerunning could reshuffle which causes dominate. The benchmark curation—excluding tasks that are not executable, not auto-evaluable, or not at least partially solvable—is reasonable for a controlled study, but it caps how far the 50% figure generalizes to real-world programmable tasks.\n\nI read the stress-test note carefully, and it holds up. The single-run issue is real and load-bearing for the cross-framework/cross-model comparisons. It does not sink the broader point that agents fail often, but it means the paper should be read as a qualitative failure map with an approximate baseline, not as a precise comparative benchmark.\n\nThis deserves serious peer review. The failure taxonomy and the case studies are useful diagnostics for the agent community. I would send it out, but I would require the authors to add multi-seed runs or exact binomial confidence intervals, report inter-annotator reliability, and make the full benchmark and evaluation harness publicly available with a commit hash. For my own work, I would cite it as evidence of common failure modes, with a caveat on the quantitative claims.","headline":"A solid failure taxonomy and a plausible 50% baseline, but the quantitative comparisons rest on single-run, small-sample data and need replication before the numbers can be trusted.","tokens_in":8515,"tokens_out":2144,"would_cite":true,"duration_ms":21281,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that current LLM-based autonomous agents complete only about half of executable coding tasks, and that the 104 observed failures split into 19 causes across planning, execution, and response generation.","keywords":["LLM agents","autonomous agents","failure analysis","failure taxonomy","benchmark","task planning","code generation","self-diagnosis"],"falsifier":"A concrete check: have independent annotators who did not write the paper re-label the published 104 failure logs into the three-tier taxonomy and measure inter-annotator agreement; low agreement would show the taxonomy is not stable. A second check is to run the same frameworks on tasks drawn without the 'at least partially solvable' filter and see whether success rates and failure proportions change sharply.","tokens_in":7517,"feed_emoji":"🤖","tokens_out":7871,"duration_ms":69811,"temperature":0.7,"pith_summary":"The paper sets out to explain why LLM-based autonomous agents fail at end-to-end programming tasks, rather than only reporting how often they succeed. It builds a benchmark of 34 executable, automatically checkable tasks drawn from web crawling, data analysis, and file operations, and runs three open-source agent frameworks with two LLM backbones. Across 204 runs, only about half of the tasks were completed exactly, and the authors classified the 104 failures into a three-tier taxonomy: planning errors, execution errors, and response-generation errors, with 19 specific causes. If this failure map is right, improving agent frameworks means targeting planning feedback and self-diagnosis, not just adding more iterations or a stronger model.","feed_headline":"Autonomous agents complete only about 50% of coding tasks","feed_subtitle":"A 34-task study traces failures to planning, execution, and response generation.","key_machinery":"The central object is the three-tier failure taxonomy built on the planner–code generator–executor architecture common to the evaluated agents. The taxonomy turns raw execution logs into a structured failure map: each failure is assigned to a task phase and then to one of 19 leaf causes, such as unrealistic planning, incorrect API usage, or exceeding the maximum round limit. This classification is what lets the paper move from a single success-rate number to targeted advice about where agent design should change.","core_discovery":"The central discovery is that current agent frameworks do not fail randomly: their failures cluster in predictable phases that line up with the planner, code generator, and executor roles inside the system. Evaluating three popular open-source frameworks with two LLM backbones on 34 tasks, the paper finds roughly a 50% exact-match completion rate, with the smaller backbone sometimes beating the larger one. The 104 failures are then organized into a three-tier taxonomy with 19 causes, from improper task decomposition and failed self-refinement in planning, through tool misuse and flawed code in execution, to context-window loss and formatting errors in response generation. The paper also reports that success improves with more allowed iterations but saturates after about ten, and that a stronger model can hurt performance when its planning steps conflict with safety constraints.","pith_inferences":["Going beyond the paper, the same three-tier taxonomy could serve as a diagnostic checklist for agent designs beyond the three frameworks tested; a testable extension is whether failure proportions shift when the planner receives explicit executor feedback after each step.","The reported 'overthinking' failures of the stronger model suggest a testable hypothesis: safety constraints and planning modules are in conflict, so letting the planner pre-check tool permissions and web-scraping rules may close part of the gap without changing the model.","The exact-match success metric is stricter than a human rubric; an editorial inference is that a partial-credit evaluation might reveal how much of the 50% failure rate is deep reasoning failure versus surface formatting failure."],"forward_implications":["Debugging an agent framework becomes a matter of locating which phase produces the failure, rather than treating the whole system as a black box.","Because success saturates after roughly ten iterations, adding more retry rounds is a weak lever; redesigning the planning loop should yield larger gains.","The smaller backbone occasionally outperforming the larger one implies that model capability and framework behavior interact, so framework evaluations should report multiple backbones.","The proposed mitigations, learning from execution feedback during replanning and an early-stop meta-controller that detects repeated failures, give concrete design targets for future agent systems."],"supporting_citations":[{"why":"One of the three agent frameworks under test; its stateful linear workflow is one of the evaluated designs.","marker":"[1]"},{"why":"One of the three agent frameworks under test; its conversation-based collaboration is one of the evaluated designs.","marker":"[2]"},{"why":"One of the three agent frameworks under test; its assembly-line multi-agent design is one of the evaluated designs.","marker":"[25]"},{"why":"The source of the data analysis tasks in the benchmark.","marker":"[22]"},{"why":"The prior evaluation practice that defines the success-rate metric used here.","marker":"[24]"},{"why":"One of the two LLM backbones; provides the stronger-model results.","marker":"[26]"},{"why":"The other LLM backbone; provides the lighter-model results that sometimes outperform the stronger model.","marker":"[27]"}],"fun_headline_variants":["Why autonomous agents fail: 50% task success rate","Agent failures cluster: planning, execution, response","Autonomous agents: 19 failure causes, 3 phases","Bigger LLM doesn't always mean better agents","Iterations help agents, but only up to 10"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 34 benchmark tasks represent real-world programmable tasks, since the authors deliberately kept only tasks that are executable, automatically checkable, and at least partially solvable by an agent; if that curation skews the sample, the 50% rate and the failure taxonomy do not generalize.","fun_headline_variants_meta":{"raw":{"variants":["Why autonomous agents fail: 50% task success rate","Agent failures cluster: planning, execution, response","Autonomous agents: 19 failure causes, 3 phases","Bigger LLM doesn't always mean better agents","Iterations help agents, but only up to 10"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"total_tokens":1325,"prompt_tokens":848,"completion_tokens":477,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":398}},"tokens_in":464,"tokens_out":477,"duration_ms":4316,"temperature":1.0,"reasoning_tokens":398,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:14:44.332697+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: have independent annotators who did not write the paper re-label the published 104 failure logs into the three-tier taxonomy and measure inter-annotator agreement; low agreement would show the taxonomy is not stable. A second check is to run the same frameworks on tasks drawn without the 'at least partially solvable' filter and see whether success rates and failure proportions change sharply.","supporting_citations":[{"cited_title":"Autogen: Enabling next-gen LLM applications via multi- agent conversations,","cited_arxiv_id":null,"evidence_quote":"One of the three agent frameworks under test; its conversation-based collaboration is one of the evaluated designs."},{"cited_title":"MetaGPT: Meta programming for a multi-agent collaborative framework,","cited_arxiv_id":null,"evidence_quote":"One of the three agent frameworks under test; its assembly-line multi-agent design is one of the evaluated designs."},{"cited_title":"Infiagent-dabench: evaluating agents on data analysis tasks,","cited_arxiv_id":null,"evidence_quote":"The source of the data analysis tasks in the benchmark."},{"cited_title":"Gpt-4o mini,","cited_arxiv_id":null,"evidence_quote":"The other LLM backbone; provides the lighter-model results that sometimes outperform the stronger model."}],"review_version":2}