{"id":"c3eb5f96-f3cb-46a9-8129-f1df7b769ba3","arxiv_id":"2504.18373","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Auto-SLURP is a new end-to-end benchmark for LLM multi-agent personal assistants, and the best current framework still fails a majority of user requests.","lead":"Auto-SLURP relabels 1,100 SLURP smart-home queries and adds simulated servers to test LLM multi-agent frameworks end to end. Current frameworks succeed on only 21% to 46% of requests.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline success-rate range and framework ranking rest on 100 unstratified test queries with no error bars; the 0.44 vs 0.46 gap between AutoGen and AgentLite is within binomial sampling noise.","rationale":"The reader's weakest_assumption was the transferability of simulated servers to real backends, which the authors themselves acknowledge in Section 6. I do not dispute that limitation, but it is a standard and acceptable feature of most agent benchmarks. The more load-bearing and internally verifiable weakness is the statistical reliability of the 100-query test set. The central claim in the abstract—'significant challenge' quantified as 21% to 46% success—depends on estimates whose sampling uncertainty is large enough to erase the observed differences between frameworks. The reader's rationale did mention '100 test queries, no error bars, unstratified sampling,' so there is partial agreement, but it was not identified as the weakest assumption. My proposed check—evaluating on a larger stratified sample and computing CIs—directly tests whether the headline numbers and rankings are stable. If they survive, the concern is resolved; if not, the paper's central empirical conclusion needs to be softened. This does not change the conditional verdict: the benchmark is promising but needs stronger statistical grounding before its quantitative claims can be relied upon.","tokens_in":9755,"tokens_out":6742,"duration_ms":69600,"concrete_test":"Score all four frameworks on a fresh stratified sample of at least 300 queries drawn from the SLURP test set, using the same relabeling and simulated-server pipeline, and compute 95% confidence intervals for each success rate and for the pairwise differences (e.g., AutoGen vs AgentLite). If the pairwise differences remain statistically significant and the rank order AgentLite > AutoGen > LangGraph > CamelAI is preserved, the small-sample concern is retired; if the intervals overlap or the order flips, the paper should report the rates as provisional and avoid ranking claims.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Auto-SLURP's central empirical claim—that current frameworks succeed on only 21% to 46% of requests—is computed from exactly 100 test queries (Section 2: '100 samples from the testing set'), with no confidence intervals, no repeated seeds, and no stratification. For n=100, the 95% Wilson interval for 0.21 is roughly [0.14, 0.30] and for 0.46 roughly [0.36, 0.56]; these intervals overlap, and the 0.44 vs 0.46 difference between AutoGen and AgentLite is well within sampling error. The paper asserts this subset 'is considered sufficient' without any power analysis, variance estimate, or demonstration that the selected 100 queries are representative of the SLURP test distribution. Because the abstract and Section 1 use these numbers to support the conclusion that Auto-SLURP poses a 'significant challenge' and Section 4.1 draws framework-level comparisons from them, the quantitative foundation of the central claim is not established. This concern is distinct from the acknowledged simulation-transfer limitation: even granting perfect simulator fidelity, the reported scores and rankings are not reliable estimates at this sample size. The benchmark may still be useful as a dataset, but its headline evaluation results are provisional until sampling variability is quantified or the test set is enlarged.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Auto-SLURP, a benchmark derived from the SLURP spoken-language-understanding dataset, in which user queries are re-labeled with enriched slot annotations and paired with simulated backend servers and external services. The goal is to support end-to-end evaluation of LLM-based multi-agent frameworks for personal-assistant scenarios, covering intent detection, task execution, and response generation. The authors evaluate four frameworks (CamelAI, LangGraph, AutoGen, AgentLite) using GPT-4 on a 100-query test subset and report successful execution rates between 21% and 46%, concluding that current state-of-the-art frameworks are substantially challenged by the benchmark. An ablation study replaces the intent agent with a fine-tuned Llama-3 8B model on the AutoGen framework and reports an improvement from an accuracy of 0.40 to 0.62. The dataset and code are publicly released on GitHub.","tokens_in":10027,"tokens_out":3749,"duration_ms":39500,"significance":"If the results are reliable, Auto-SLURP fills a genuine gap: existing benchmarks largely evaluate individual LLMs or single-agent tool use, whereas Auto-SLURP attempts to measure entire multi-agent orchestration pipelines end to end. The public release of the dataset, the deterministic temperature-zero evaluation protocol, and the explicit workflow definition are strengths that should facilitate reuse. However, the quantitative evidence base is thin: the headline success rates come from exactly 100 test queries with no confidence intervals, no stratification, and no power analysis, and the differences between frameworks are within binomial sampling error. The dataset-construction process also lacks documented quality assurance. Because the central claim that Auto-SLURP is a significant challenge for current frameworks rests on these numbers, the paper needs additional statistical grounding or more modest framing before the result can be considered established.","major_comments":[{"comment":"The central quantitative claim is based on only 100 test samples, selected uniformly at random from the SLURP test set, with no confidence intervals, no repeated seeds, and no power analysis. For n=100, the 95% Wilson interval for the reported success rate of 0.21 is approximately [0.14, 0.30] and for 0.46 is approximately [0.36, 0.56]; these intervals overlap substantially, so the ranking between AutoGen (0.44) and AgentLite (0.46) is well within sampling noise. The statement in Section 2 that this subset 'is considered sufficient' is not supported by any variance estimate or by a comparison of the subset's intent/domain distribution to the full SLURP test distribution. I recommend that the authors either enlarge the test set, report confidence intervals and ideally stratified sampling, or substantially soften the claim that the reported numbers establish a significant challenge and a meaningful framework ranking.","section":"Section 2 and Table 2"},{"comment":"The automated evaluation tool is described only as 'consistent and efficient,' but its decision rules are not specified. In particular, the failure attribution criteria in Appendix B include subjective conditions such as 'if the URL agent receives an incorrect intent but is capable of correcting it and fails to do so,' with no operational definition of 'capable.' There is no inter-annotator agreement, no validation that the attribution matches human judgment, and no discussion of how partial failures or cascading errors are assigned to a single agent. This matters because Section 4.1 uses Table 3 to conclude that 'the main source of failure stems from the intent agent,' and Section 4.2 builds an ablation on that conclusion. Without a documented and validated attribution procedure, the failure analysis is not reproducible.","section":"Section 3.3 and Appendix B"},{"comment":"The dataset is the primary contribution, yet its construction is described with a single example and no annotation guidelines, no description of how the new slots were derived from the original SLURP annotations, no quality-control measures, and no report of annotator agreement. The enriched slots must be correct and complete for the end-to-end evaluation to be meaningful, because a wrong or missing slot annotation can cause a framework to fail through no fault of the framework. I recommend providing a detailed annotation protocol, example counts of slot types, and a reliability analysis, or at least a manual audit of a sample of the 1,000 training and 100 test annotations.","section":"Section 2, Table 1"},{"comment":"The paper acknowledges that simulated servers may not fully mimic real systems, but it does not address the reproducibility risk of relying on live third-party APIs for domains such as weather, search, and news. Because these APIs return time-varying content and may change their interfaces, two runs of the benchmark at different times can yield different outcomes even with temperature set to zero. The authors should state whether API responses were cached or fixed, provide versioned snapshots of external service responses, or otherwise document how reproducibility is ensured. Without this, the 'automated evaluation tool' does not guarantee consistent comparison across framework versions or research groups.","section":"Section 6 and Section 3.1"}],"minor_comments":[{"comment":"The sentence 'We assess the end-to-end execution tasks, not just the nature language understanding tasks' contains a typo: 'nature' should be 'natural.'","section":"Section 5"},{"comment":"The caption says 'AutoGen original finetuned' but the column header is 'funetuning'; correct the typo and clarify that 'original' refers to the original Llama-3 8B model, not the GPT-4-based AutoGen configuration in Table 2.","section":"Table 5 caption"},{"comment":"The metric is defined as 'successful execution rate' in Section 3.3, but Table 2 labels it 'acc' and Section 4.1 calls it 'accuracy score.' Use a single consistent term to avoid confusion.","section":"Table 2 and Section 3.3"},{"comment":"In the first paragraph of Section 1, 'SocialBench (Chen et al., 2024)' is cited, but the reference list entry 'Chen et al. 2024' is titled 'Roleinteract: Evaluating the social interaction of role-playing agents,' which does not match the cited benchmark name; please correct the reference. Also, 'ALFWorld (Shridhar et al.)' lacks a year and venue in the citation.","section":"Introduction and References"},{"comment":"The domain-specific accuracy tables exclude domains with fewer than three samples, but they do not report the number of test samples per domain. For a 100-query test set, many domains likely contain very few samples, so percentages such as 0.0% versus 66.7% may represent one or two examples; reporting per-domain counts and confidence intervals would prevent overinterpretation.","section":"Appendix C, Tables 6 and 7"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable dataset contribution, but the experimental evidence for the headline claim is statistically fragile, and the dataset-construction process needs more transparency. The authors should be pushed to either provide a substantially larger or stratified test sample with confidence intervals or explicitly reframe the results as a small-scale demonstration rather than a definitive benchmark evaluation. The reference list also contains several informal citations (e.g., LangGraph 2023, LangChain 2022 without authors or venues) that should be cleaned up before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: Auto-SLURP is a real contribution to a thin area—there is no standard end-to-end benchmark for LLM-based multi-agent personal assistants. The authors take SLURP, relabel slots to include execution-relevant info, add simulated servers and external APIs, and define a workflow with specialized agents. That is a legitimate and useful artifact, and the paper is honestly written. The failure attribution criteria in Appendix B are a nice touch, and the ablation with a fine-tuned Llama intent agent is coherent.\n\nBut the headline claim—that current state-of-the-art frameworks succeed on only 21% to 46% of requests—needs much stronger support than it gets. The numbers come from exactly 100 test queries (Section 2), with no confidence intervals, no repeated seeds, no stratification. At n=100, the 95% Wilson intervals for 0.21 and 0.46 are roughly [0.14, 0.30] and [0.36, 0.56]; they overlap, and the AutoGen vs. AgentLite difference (0.44 vs. 0.46) is pure noise. The paper's assertion that the subset is 'sufficient' is not backed by any power analysis or representativeness check.\n\nThe authors do acknowledge that simulated servers may not transfer to real systems, and that is a fair limitation. But even granting full simulator fidelity, the reported rankings just do not support the conclusion that Auto-SLURP is a 'significant challenge'—we don't know the precision of those rates. That is a separate and more serious problem for the paper's argument.\n\nWhat would fix it: enlarge the test set to a few hundred or thousand queries, stratify by domain, report error bars (or do multiple runs with different seeds), and describe the automated evaluation tool's decision rules. The benchmark itself may well be useful after that; the dataset and code availability are points in its favor.\n\nWho is this for? Researchers building multi-agent frameworks or personal assistants who need a shared yardstick. With the current evaluation section, it is a decent dataset paper in need of revision, not a reliable comparison of frameworks. I would send it to peer review, but I would tell the authors to redo the evaluation before it goes out again.","headline":"Useful benchmark artifact, but the headline success rates rest on 100 test queries with no error bars, so the rankings are provisional.","tokens_in":10516,"tokens_out":2621,"would_cite":true,"duration_ms":25483,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Auto-SLURP evaluates multi-agent assistants from intent to execution and finds even the best framework completes only 46% of requests.","keywords":["Auto-SLURP","multi-agent frameworks","smart personal assistant","end-to-end evaluation","LLM agents","SLURP","intent detection","benchmark dataset"],"falsifier":"Deploy the same four frameworks against the real smart-home services that the simulated servers approximate, such as real calendar, email, weather, and IoT APIs, and compare end-to-end success rates. If the ranking of frameworks changes materially, for example if AgentLite no longer leads, then scores on Auto-SLURP do not predict live assistant performance.","tokens_in":9558,"feed_emoji":"🤖","tokens_out":8992,"duration_ms":72979,"temperature":0.7,"pith_summary":"Auto-SLURP is a benchmark that turns the SLURP smart-home assistant dataset into an end-to-end test for LLM-based multi-agent frameworks. The authors relabel SLURP queries with the slot information needed to actually execute them and attach simulated backend servers and external APIs, so a framework must not only understand the user's request but also orchestrate specialized agents to perform it and return an answer. They report that four representative frameworks complete only 21% to 46% of requests successfully, with intent prediction the largest source of failure. The point of the paper is that multi-agent assistant frameworks should be evaluated on the whole pipeline, not just on natural language understanding, and that current frameworks are far from reliable enough for this task.","feed_headline":"Auto-SLURP benchmark: top framework completes just 46%","feed_subtitle":"A new benchmark sends commands through simulated smart-home servers; even the best framework fails over half the time.","key_machinery":"The load-bearing mechanism is the simulated end-to-end execution pipeline. Each of 23 domains gets a dedicated simulated server, with external APIs for search, weather, and news, and every query is relabeled with the slots needed to run it. Frameworks must implement a fixed workflow: a program manager agent routes the request to intent, time, location, and URL agents, then a request agent calls the server and a response is generated. Success is scored by whether the entire pipeline completes correctly, which makes the benchmark measure coordination and tool use rather than classification alone.","core_discovery":"The paper's central claim is that Auto-SLURP, built by extending SLURP's original intent-slot annotations and pairing them with simulated execution servers and external services, provides a valid end-to-end measure of multi-agent framework performance for smart personal assistants. On this measure, the best framework, AgentLite, succeeds on 46% of 100 test queries, while CamelAI, LangGraph, and AutoGen succeed on 21%, 32%, and 44% respectively. Error analysis attributes most failures to the intent agent. A finetuned Llama 3 8B intent agent raises AutoGen's end-to-end success from 40% to 62%, a 55% relative improvement, which the paper reads as evidence that component-level weaknesses, rather than framework-level orchestration alone, limit current assistants.","pith_inferences":["A testable extension would inject realistic backend failures, such as timeouts, malformed responses, or authentication errors, into the simulated servers; if rankings change under injected noise, robustness to real-world server behavior is an additional axis the current metric does not capture.","The fixed workflow of one program manager plus specialized agents may favor frameworks that support role-specific prompts; a fairer comparison could let each framework define its own workflow and still measure only end-to-end success.","Because the data are English smart-home commands from SLURP, generalizing to other languages or enterprise domains would require rebuilding the simulated servers, so the current results should not be read as a statement about assistant performance outside that setting."],"forward_implications":["If Auto-SLURP measures what it claims, framework design choices—how prompts are separated, how agents are orchestrated, whether reasoning steps are inserted—directly change end-to-end success rates, so the benchmark can compare frameworks on equal footing.","Because intent prediction is the dominant failure point, improving the intent component, for example by finetuning a smaller open model, is a concrete lever for whole-system gains; the paper's 55% relative improvement on AutoGen demonstrates this.","The benchmark's cost figures, from 0.14 to 0.80 USD per query across frameworks, make it possible to compare accuracy against operating cost for multi-agent assistant deployments.","With only 1,000 training and 100 test queries, Auto-SLURP is lightweight enough to be used for finetuning and rapid iteration in academic settings."],"supporting_citations":[{"why":"Source of the original SLURP queries and intents that Auto-SLURP relabels with execution-ready slots.","marker":"Bastianelli et al., 2020"},{"why":"Earlier benchmarking of SLURP as a spoken language understanding resource that Auto-SLURP extends beyond understanding to execution.","marker":"Liu et al., 2021"},{"why":"Defines CamelAI, one of the multi-agent frameworks whose end-to-end success rate is measured.","marker":"Li et al., 2023"},{"why":"Defines AutoGen, the framework used in the finetuning ablation and a comparison baseline.","marker":"Wu et al., 2023"},{"why":"Defines AgentLite, the best-performing framework in the benchmark.","marker":"Liu et al., 2024"},{"why":"The graph-based framework included as a comparison baseline.","marker":"LangGraph (2023)"},{"why":"Supplies GPT-4, the LLM backbone used for all framework runs.","marker":"Achiam et al., 2023"},{"why":"Provides the Llama 3 8B model finetuned for the intent-agent ablation.","marker":"AI@Meta, 2024"}],"fun_headline_variants":["Best multi-agent framework fails 54% on new Auto-SLURP benchmark","Auto-SLURP benchmark: even best agents fail over half the time","Fine-tune intent agent boosts AutoGen from 40% to 62%","New benchmark: top LLM multi-agent framework only 46% accurate","Auto-SLURP challenge: AgentLite best at 46%, AutoGen 44%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's scores transfer to real assistants only if the simulated servers and third-party APIs behave like production backends; the authors themselves flag that simulated services may not fully mimic real-world systems.","fun_headline_variants_meta":{"raw":{"variants":["Best multi-agent framework fails 54% on new Auto-SLURP benchmark","Auto-SLURP benchmark: even best agents fail over half the time","Fine-tune intent agent boosts AutoGen from 40% to 62%","New benchmark: top LLM multi-agent framework only 46% accurate","Auto-SLURP challenge: AgentLite best at 46%, AutoGen 44%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001982,"raw_usage":{"total_tokens":7706,"prompt_tokens":877,"completion_tokens":6829,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":6722}},"tokens_in":493,"tokens_out":6829,"duration_ms":48583,"temperature":1.0,"reasoning_tokens":6722,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:17:12.116852+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Deploy the same four frameworks against the real smart-home services that the simulated servers approximate, such as real calendar, email, weather, and IoT APIs, and compare end-to-end success rates. If the ranking of frameworks changes materially, for example if AgentLite no longer leads, then scores on Auto-SLURP do not predict live assistant performance.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Earlier benchmarking of SLURP as a spoken language understanding resource that Auto-SLURP extends beyond understanding to execution."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The graph-based framework included as a comparison baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies GPT-4, the LLM backbone used for all framework runs."}],"review_version":1}