{"id":"33596820-14a8-4130-b178-f8d3f7caa366","arxiv_id":"2508.05002","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A feedback-driven multi-agent system with plan validation and smart memory claims to outperform prior NL-to-analytics systems on heterogeneous benchmarks, though its evaluation is problematic.","lead":"AgenticData is a multi-agent system that turns natural-language questions into data-analysis plans that run over both databases and unstructured files. The authors report large accuracy and cost wins over prior systems on three benchmarks, but the evaluation has serious methodological gaps.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported DABStep/Spider gains may reflect cross-task memory leakage: long-term memory persists across benchmark tasks and stores task-specific plans, but the paper never states it is reset per query.","rationale":"I read the paper as a systems contribution: the multi-agent planning, validator feedback, and cost-aware optimization are plausible and the Wikipedia/Palimpzest comparison at least pairs the same LLM. However, the headline accuracy claim is the load-bearing result, and it is not secure because the memory subsystem creates a concrete contamination channel. The reader's weakest assumption—that long-term memory is not fed from the benchmark test set during evaluation—is exactly the right concern. Figure 2 strengthens it by showing task-specific plans stored as 'Historical General Good Cases', and Section 2.2 confirms that the long-term memory is designed to persist and be retrieved for similar tasks. The paper provides no ablation, no memory-reset statement, and no fresh-database guarantee for the DABStep or Spider-2.0-Lite runs. This is not a matter of disagreement with existing consensus; it is an internal design feature whose evaluation protocol is unspecified. A single controlled rerun with memory cleared would settle the issue. Since the paper as submitted does not rule out leakage, the reader's REJECT verdict is appropriate, and I would not change it. If the authors provide the memory-reset experiment and the result holds, the claim could be revisited.","tokens_in":12409,"tokens_out":3524,"duration_ms":43662,"concrete_test":"Rerun the DABStep evaluation with long-term memory disabled or cleared after every task, keeping all other components (same Qwen3 model, same prompts, same validator/optimizer settings) unchanged. If the hard-task accuracy remains near 50.79%, the concern is resolved; if it drops materially toward the stateless baselines, the reported gains are inflated by cross-task memory. As a secondary check, inspect the memory table after the original run: if entries have task embeddings matching later DABStep query text, that confirms the leakage path.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—AgenticData significantly outperforms SOTA on DABStep (94.44%/50.79% vs. 80.56%/41.01%/81.94%/19.84%)—is only valid if each benchmark task is solved from scratch. The memory design undermines this requirement. Section 2.2 says long-term memory is stored in a vector DB relation table with attributes [task, task-embedding, memory], and Figure 2 shows long-term memory containing 'Historical General Good Cases'—a complete worked plan for a specific fee-calculation task, not just generic syntax rules. During planning, the system retrieves knowledge of similar tasks from this table. If the 450 DABStep tasks are processed in one session without clearing memory, later tasks can benefit from plans, error messages, and corrections derived from earlier test questions. The paper never states that memory is reset per task, that each query uses a fresh vector DB, or that knowledge acquired during evaluation is excluded. Baselines like Amity and MultiStep do not share this cross-task memory, so the reported accuracy gap could be an artifact of test-set contamination. The cost/latency advantage (Figures 5–6) is coupled to the same mechanism, since fewer iterations on hard tasks may reflect prior exposure to similar benchmark problems.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AgenticData, a multi-agent natural-language analytics system for heterogeneous structured and unstructured data. The system converts an NL query into a semantic plan using a data profiling agent, a planning agent, a manipulation agent, a plan validator, a memory manager, and a cost/quality optimizer. The authors claim state-of-the-art accuracy on DABStep, Spider-2.0-Lite, and Wikipedia benchmarks, as well as lower LLM cost and latency than strong baselines. The main contributions are the feedback-driven planning framework, the multi-agent collaboration with persistent memory, and the semantic query optimization model.","tokens_in":12719,"tokens_out":4129,"duration_ms":46878,"significance":"If the reported results are valid, AgenticData would be a meaningful advance in NL-driven data analytics over heterogeneous data, especially because it claims to beat hand-coded pipelines and other agents on public benchmarks while reducing cost. The paper provides a detailed system description and evaluates against several leaderboard baselines. However, the significance is heavily conditional on the soundness of the experimental evaluation, and the evaluation as presented has load-bearing flaws that prevent the reader from trusting the headline accuracy and cost claims.","major_comments":[{"comment":"The long-term memory is a persistent vector-database relation with attributes [task, task-embedding, memory] and is retrieved for 'similar tasks' during planning. Figure 2 shows that the stored 'Historical General Good Cases' includes a complete worked plan for a specific fee-calculation query. The paper never states that memory is reset per benchmark task, per query, or after each evaluation. Since DABStep has 450 tasks and other benchmarks have hundreds of queries, later tasks can benefit from plans, error messages, and corrections derived from earlier test questions. Baselines such as Amity and MultiStep do not share this cross-task memory. Therefore the accuracy advantages in Table 1 (94.44%/50.79% vs. 80.56%/41.01%/81.94%/19.84%) may be an artifact of test-set contamination rather than intrinsic capability. The paper must either state and enforce a memory-reset protocol or report re","section":"Section 4, Step 3; Figure 7"},{"comment":"The cost-optimality evaluation is circular. The 'quality score' used to guide LLM selection and to determine the 'Optimal Pareto Frontier' in Figure 7 is produced by the system's own internal plan-quality model. If that same model defines both the optimization objective and the frontier against which AgenticData is compared, Figure 7 merely demonstrates that the optimizer is consistent with its own cost model, not that it achieves Pareto-optimal real-world cost/quality trade-offs. No external ground-truth quality labels or human evaluation are provided to calibrate the internal quality model. The paper should validate the quality model against actual answer accuracy on the Wikipedia benchmark, or compare against a frontier derived from external quality measures.","section":"Tables 1 and 2; Section 5.2"},{"comment":"The accuracy comparisons are not controlled: AgenticData uses Qwen3 while the DABStep baselines use Gemini2.5 and GPT-o3, and the Spider-2.0-Lite baseline also uses Qwen3. Differences in the underlying LLM are a known major confound in agent-system evaluation. The paper should either use the same LLM backbone across all systems or include an ablation that isolates the effect of the multi-agent framework from the choice of the base model. Without this, the 'significantly outperforming' claim is not attributable to the proposed architecture.","section":"Section 5.2"},{"comment":"The number-of-steps comparison (Figures 5 and 6) is coupled to the same memory-leak concern: a lower step count on later tasks may simply reflect that the system has already seen similar plans and error corrections from earlier test queries. Additionally, no error bars, confidence intervals, or multiple runs are reported for any accuracy or cost figure, so the statistical reliability of the claimed improvements cannot be assessed.","section":"Figures 3-6"}],"minor_comments":[{"comment":"The environment description is confusing: 'All systems are run on a laptop featuring an Apple M3 chip' while also stating that LLMs are executed on a server with 8 H200 GPUs. Please clarify which components run where.","section":"Figure 1"},{"comment":"Typographical error: 'UNIOIN' should be 'UNION'. Also 'Summarized Common Knowledges' in Figure 2 should be 'Summarized Common Knowledge'.","section":"Figure 2"},{"comment":"The baselines Amity and MultiStep are referenced only as leaderboard entries; provide citations or URLs to the system descriptions. Palimpzest and ReFoRCE are cited, but the exact versions/configurations used should be specified.","section":"Section 4"},{"comment":"The stopping criterion for quality degradation is given as 'less than a constant value, such as 0.05'. It is unclear whether this constant is a free parameter tuned on the benchmarks and how sensitive the results are to it.","section":"Section 5.1"},{"comment":"The Wikipedia benchmark description says '1000 randomly selected web pages' but only '100 queries from Stack Exchange'. Clarify the exact evaluation set size and whether all 100 queries were used.","section":"Section 5.1"}],"recommendation":"reject","confidential_remarks":"The paper has a plausible system design, but the two main empirical claims are undermined by methodology problems: potential test-set contamination through persistent long-term memory, and a circular cost/quality evaluation. These are load-bearing, not cosmetic. Even if the memory issue could be fixed by rerunning with memory reset, the cost-optimality claim requires a fundamentally different evaluation design with external quality labels. As written, the evidence does not support the abstract's claim of 'significantly outperforming state-of-the-art methods'."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The architecture is the real contribution here. The three-agent planning loop (profiling, planning, manipulation) with grammar/semantic validation and the smart-memory routing scheme is a genuinely new integration. The memory manager's two-phase selection and the plan-level quality model for cost optimization are worth a close look. The system is engineering in the good sense: it connects the right known pieces (semantic operators, vector retrieval, LLM cascades, MCP) in a way that could plausibly help real users do heterogeneous data analytics without writing code. The DABStep plan example and the detailed memory design show real systems thinking. I believe the authors know what they are building.\n\nThe soft spots are in the evaluation, and they are load-bearing. First, the headline tables compare against leaderboard results that use different LLMs (Gemini2.5, GPT-o3) than AgenticData's Qwen3. Leaderboard numbers are not a controlled baseline; a 94% vs 80% gap on DABStep easy tasks is not interpretable without running the same pipeline on the same model. Second, the long-term memory design is a real contamination risk. The memory stores \"historical general good cases\" with task-specific plans, and retrieval is by task embedding. There is no statement that memory is cleared or isolated per benchmark task. On a 450-task benchmark, later tasks could benefit from earlier ones—leaving baselines that do not have this memory at a systematic disadvantage. This is not a manufactured concern; the paper's own Figure 2 shows a complete worked plan in long-term memory. Third, the Wikipedia cost-optimality claim in Figure 7 is circular: the same plan-quality model defines the optimizer's objective and the \"optimal Pareto frontier\" used for validation. That figure shows the system is close to its own internal notion of optimal, which is not evidence of real optimality.\n\nWhat is missing are ablations. The memory routing, the multi-agent validator, and the cost model are each substantial design choices, and none are turned off to measure their contribution. Error bars and per-query cost numbers (not just iteration counts) would also help.\n\nThe paper is serious and the architecture deserves attention. But as submitted, the central empirical claims are not established. This is a revise-and-resubmit situation, not a desk reject and not an accept. If the memory contamination path is closed and the baselines are run under controlled conditions, this could be a solid systems paper. For now, treat the accuracy numbers as upper bounds.\n\nRecommendation: send it to peer review, but expect major revision. I would not cite the current numbers in my own work until the evaluation is cleaned up.","headline":"A serious systems paper whose architecture is thoughtful but whose headline accuracy/cost claims are not yet supported by the experimental evidence as reported.","tokens_in":13188,"tokens_out":648,"would_cite":false,"duration_ms":10364,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AgenticData turns natural-language questions into semantic plans that beat hand-coded analytics pipelines on three benchmarks, the paper reports.","keywords":["agentic data analytics","natural language query","semantic operators","multi-agent planning","LLM cost optimization","semantic plan validation","memory management","heterogeneous data"],"falsifier":"Run AgenticData on the DABStep benchmark with the long-term memory cleared before every task (or disabled entirely) and compare accuracies. If the gap to leaderboard baselines collapses or reverses, the performance claim depends on cross-task memory leakage rather than on the planner itself.","tokens_in":12325,"feed_emoji":"📊","tokens_out":2051,"duration_ms":25658,"temperature":0.7,"pith_summary":"The paper introduces AgenticData, a system that lets users ask data-analysis questions in plain language and get answers from heterogeneous, partially unstructured data without a human writing code or managing a pipeline. The system's core move is to convert each natural-language query into a semantic plan built from relational and semantic operators, using a team of specialized agents (data profiling, data planning, data manipulation) that collaborate through feedback loops and a persistent memory of past errors and patterns. The authors claim this approach reaches higher accuracy than existing leaderboard systems on DABStep, Spider-2.0-Lite, and Wikipedia benchmarks, while also reducing LLM costs. A sympathetic reader would care because it suggests a practical path from expert-coded semantic analytics to autonomous, NL-driven analytics over messy real-world data.","feed_headline":"NL queries beat coded pipelines on hard data tasks","feed_subtitle":"AgenticData reports 50.8% on hard DABStep vs. 41.0% for the top leaderboard coding agent, at lower LLM cost.","key_machinery":"The central mechanism is the semantic plan: a tree-structured logical plan combining relational operators (Scan, Filter, Project, Join, Aggregate, Union, Merge) with semantic operators (Extract, Sentiment Analysis, Semantic Filter, Semantic Group, Semantic Join). The plan is generated by a multi-agent planner and validated by a two-phase validator; optimization then applies cost rules, dynamic-programming join ordering, and a quality-aware LLM cascade to reduce LLM invocation cost. The memory manager stores error messages and recurring patterns in a vector database, retrieving them for analogous future tasks.","core_discovery":"AgenticData claims to be the first agentic data analytics system that directly analyzes both unstructured and structured data from natural-language queries, without requiring users to write code or to pre-extract a structured schema. It generates a logical semantic plan by coordinating a data profiling agent that discovers and describes datasets, a data planning agent that decomposes the task and selects data, and a data manipulation agent that fills in operator details. A validator cross-checks the plan semantically and grammatically, feeding corrections back through a memory manager that accumulates short-term context and long-term knowledge across tasks. The paper reports 94.44% accuracy","pith_inferences":["The reported accuracy gains may partly stem from the memory manager retaining knowledge across benchmark tasks; if so, the gains could shrink when memory is reset per task, and the comparison against stateless baselines is not apples-to-apples. This is an editorial concern, not stated in the paper.","The semantic-plan cost model and cardinality estimation framework could be extracted and reused by other NL2Data systems, independent of AgenticData's specific agents.","A natural testable extension is to evaluate AgenticData with memory disabled or cleared per task to isolate the contribution of long-term knowledge.","The paper's architecture suggests a general recipe for turning any analytical benchmark into an agentic language task, which could accelerate progress on heterogeneous data analysis but also demands careful benchmark hygiene to avoid test-set contamination via persistent memory."],"forward_implications":["If the reported accuracy holds, natural-language interfaces could replace expert-coded semantic pipelines for many real-world analytics tasks, lowering the barrier for non-programmers.","The feedback-driven multi-agent planning with memory could be extended to more data domains and heterogeneous sources beyond the financial and Wikipedia benchmarks tested.","The cost model and LLM cascade selection could generalize to other semantic query processing systems, making AI-powered analytics more affordable.","The system's support for non-predefined operators via code generation suggests a path toward open-ended analytics beyond fixed operator sets.","The memory mechanism implies that the system improves with experience, potentially enabling continual learning as it is used over time."],"supporting_citations":[{"why":"Palimpzest: the semantic-operator baseline that AgenticData outperforms on Wikipedia, and the source of the semantic operator paradigm.","marker":"[7]"},{"why":"Semantic operators: defines the declarative model for AI-based analytics over text that AgenticData builds on.","marker":"[9]"},{"why":"AOP: represents the single-step LLM pipeline orchestration approach that AgenticData contrasts with its multi-agent planning.","marker":"[15]"},{"why":"Chat2Data: an interactive NL2Data system with RAG and vector databases, serving as a baseline for single-step NL-to-data planning.","marker":"[17]"},{"why":"Survey of text-to-SQL: establishes the limitation that NL2SQL works only on structured databases, motivating AgenticData's broader scope.","marker":"[8]"},{"why":"TAG: an AI-and-database unification approach that informs the semantic-plan and optimization ideas.","marker":"[2]"},{"why":"GaussML: supplies the selectivity estimation method used in join-order optimization.","marker":"[5]"},{"why":"GaussDB-vector: the vector database infrastructure used to store the memory and semantic catalog.","marker":"[13]"}],"fun_headline_variants":["Ask questions, get answers: AgenticData tames messy data","No code, just questions: AgenticData crunches unstructured data","AgenticData lets you query messy data in plain English","Natural language to data insights: AgenticData beats coders","Skip the code: AgenticData answers NL queries on mixed data"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The central claim assumes that the long-term memory vector database is not populated from the benchmark test set during evaluation, so that reported accuracy reflects each task being solved without knowledge of other test queries.","fun_headline_variants_meta":{"raw":{"variants":["Ask questions, get answers: AgenticData tames messy data","No code, just questions: AgenticData crunches unstructured data","AgenticData lets you query messy data in plain English","Natural language to data insights: AgenticData beats coders","Skip the code: AgenticData answers NL queries on mixed data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000864,"raw_usage":{"total_tokens":3556,"prompt_tokens":693,"completion_tokens":2863,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":437,"completion_tokens_details":{"reasoning_tokens":2775}},"tokens_in":437,"tokens_out":2863,"duration_ms":22006,"temperature":1.0,"reasoning_tokens":2775,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T23:37:21.190692+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run AgenticData on the DABStep benchmark with the long-term memory cleared before every task (or disabled entirely) and compare accuracies. If the gap to leaderboard baselines collapses or reverses, the performance claim depends on cross-task memory leakage rather than on the planner itself.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Palimpzest: the semantic-operator baseline that AgenticData outperforms on Wikipedia, and the source of the semantic operator paradigm."},{"cited_title":"Patel, S","cited_arxiv_id":null,"evidence_quote":"Semantic operators: defines the declarative model for AI-based analytics over text that AgenticData builds on."},{"cited_title":"Wang and G","cited_arxiv_id":null,"evidence_quote":"AOP: represents the single-step LLM pipeline orchestration approach that AgenticData contrasts with its multi-agent planning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Chat2Data: an interactive NL2Data system with RAG and vector databases, serving as a baseline for single-step NL-to-data planning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Survey of text-to-SQL: establishes the limitation that NL2SQL works only on structured databases, motivating AgenticData's broader scope."},{"cited_title":"Biswal, L","cited_arxiv_id":null,"evidence_quote":"TAG: an AI-and-database unification approach that informs the semantic-plan and optimization ideas."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GaussML: supplies the selectivity estimation method used in join-order optimization."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GaussDB-vector: the vector database infrastructure used to store the memory and semantic catalog."}],"review_version":1}