REVIEW 3 major objections 3 minor 2 cited by
This paper argues that AI-driven analytics needs a runtime merging Deep Research agents' flexible planning with semantic operators' cost-based optimized execution, and presents a prototype achieving 1.95x better F1 and up to 76.8% cost savi
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
A prototype combining Deep Research planning with Palimpzest-style semantic operator optimization beats open-code agents on two unstructured analytics queries.
T0 review reviewed 2026-08-05 challenge →
load-bearing objection A clear and honest vision paper with a working prototype that combines Deep Research agents with semantic operator optimization; the synthesis is worth discussing, but the measured efficiency savings rely on an untuned baseline and only two queries. the 3 major comments →
Deep Research is the New Analytics System: Towards Building the Runtime for AI-Driven Analytics
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
At the center of the paper is a claim about the right division of labor: rather than choosing between an agent that writes ad hoc Python and a compiler that executes fixed semantic operators, let the agent write a semantic-operator program and then hand that program to a cost-based optimizer. The authors implement this by adding two logical operators, compute and search, to Palimpzest. Each is physically realized as a CodeAgent, and each gets a tool that can execute a natural-language instruction as an optimized Palimpzest program. A new Context abstraction gives agents access to index-based lookups, custom tools, and a natural-language description of the data, while a ContextManager caches
What carries the argument
The load-bearing object is the Context, an extension of Palimpzest's Dataset that adds a natural-language description, user-defined index/top-k access methods, and custom tools. Around it sit two agent-implemented logical operators—compute, which generates a specific output, and search, which retrieves information to enrich a Context's description—each with a tool that emits an optimized semantic-operator program. The optimizer then takes over, selecting cheaper models and pruning redundant work, while the ContextManager treats each materialized Context like a database view and reuses it for similar future instructions.
Load-bearing premise
The headline savings are measured against a CodeAgent+ baseline that frequently runs several semantic filters without checking the output of the first one; with better prompting that baseline could become much stronger, and only two queries were tested, so the reported 76.8% and 72.7% may not generalize.
What would settle it
Run the two evaluation queries, plus a larger sample from Kramabench, against a CodeAgent+ variant whose system prompt explicitly requires inspecting the result of each semantic filter before invoking the next, and compare cost, latency, and F1. If the gap narrows to near zero, the savings come mostly from baseline weakness rather than the runtime's optimizations.
If this is right
- If correct, Deep Research agents can be recast as query writers whose output is a compiled plan, giving them the same cost and latency predictability as database engines.
- Semantic-operator systems gain the ability to handle interactive, multi-step queries that require reasoning across files and iterative refinement, not just fixed pipelines.
- Materialized Contexts create a natural cross-query cache: after one query enriches a data lake, later queries can reuse that enriched view instead of reprocessing raw files.
- Cost-based model selection can be applied inside agent-written programs—a cheap model for simple filters, a stronger model only where needed—without the agent micromanaging the choice.
- The search operator provides a runtime fallback when a compute step fails or a Context is too thin, so plans can be repaired dynamically.
Where Pith is reading between the lines
- Editorial inference: if the baseline were strengthened to check intermediate operator outputs, the reported 76.8% and 72.7% cost-and-latency advantages could shrink; the savings are a proof of concept, not a guaranteed bound.
- Editorial inference: the same agent-writes-optimized-program pattern could be applied to other semantic-operator engines, not just Palimpzest, since the agent only needs a tool interface that accepts a natural-language instruction and returns optimized execution.
- Editorial inference: over many queries, the ContextManager's cache could learn a data lake's structure and make later queries dramatically cheaper—an effect not measured here because only two queries were run.
- Editorial inference: the strongest testable prediction is that on a broader benchmark the quality gap between this prototype and a well-prompted CodeAgent+ will be smaller than the cost gap; a reader should look for variance and more queries before trusting the headline numbers.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that neither semantic-operator systems nor Deep Research agents alone are sufficient for AI-driven analytics over unstructured data, and proposes a runtime that combines the optimized execution of semantic operators with the flexibility of Deep Research agents. The prototype extends Palimpzest with compute and search logical operators physically implemented as CodeAgents, a Context abstraction with indexing/tools, and a ContextManager for materialized-context reuse. The evaluation reports two queries: a Kramabench legal query where the prototype achieves 0.02% error versus 17% for a handcrafted semantic-operator program, and an Enron query where the prototype matches CodeAgent+ quality (98.67% F1) while saving 76.8% cost and 72.7% runtime, and improves F1 by 1.95x over a plain CodeAgent. The vision is clear and the prototype is concrete, but the empirical evidence is narrow: two hand-picked queries, three trials each, no variance reporting, and a CodeAgent+ baseline whose inefficiency is acknowledged in the text.
Significance. If the reported results hold, the paper makes a useful first step toward unifying Deep Research planning with cost-based semantic-operator optimization. The numerical claims in Tables 1 and 2 are internally consistent: 98.67/50.53 is indeed 1.95x, and the cost/runtime savings relative to CodeAgent+ are 76.8% and 72.7%. The paper also introduces abstractions (compute/search operators, Context, ContextManager) that are likely to influence follow-on system work. However, the significance is currently limited by the evaluation: the central efficiency claim is demonstrated against a baseline that the paper itself describes as inefficient, and there is no variance analysis. The reader cannot yet tell whether the advertised savings are a property of the proposed runtime or an artifact of the untuned baseline.
major comments (3)
- [Section 4, Table 2] The central efficiency claim (76.8% cost and 72.7% runtime savings over CodeAgent+) is not yet robust. The paper itself states that CodeAgent+ 'often executed multiple semantic filters in sequence without checking the output of the first semantic filter' and that Palimpzest's optimizer 'was able to use cheaper models for some of the semantic operators.' The savings therefore conflate the compute operator's program-writing ability, the optimizer's model selection, and the baseline's lack of output-gated filter invocation. A competitively prompted CodeAgent+, or one given an optimize_and_run tool, could close much of this gap. Please ablate these factors and report per-trial costs/runtimes; as written, the abstract's efficiency claim is stronger than the evidence.
- [Section 4, Tables 1-2] The empirical support is two hand-picked queries, each run three times, with only averages and no variance. LLM cost and latency are highly variable across runs, so the headline 76.8%/72.7% savings may be dominated by a single trial. The paper should report per-trial values, ranges, or error bars, and ideally include additional Kramabench queries. Without this, the quantitative claims in the abstract are not fully supportable.
- [Section 4, Table 2] The 1.95x F1 improvement over CodeAgent is presented without the corresponding cost/latency tradeoff: CodeAgent runs in 37s at $0.08, while PZ compute runs in 546.2s at $1.17. The quality claim is true but incomplete. Please state explicitly that the F1 gain comes at roughly 15x runtime and 11x cost, and clarify whether the comparison is intended as quality-at-fixed-budget or as a different operating point. This is important for a fair interpretation of the contributions.
minor comments (3)
- [Section 2.4 and Section 3] The ContextManager and materialized-Context reuse are described as a contribution, but no experiment isolates their effect. If this is future work, please state that explicitly; if it is meant to support the runtime vision, provide at least a preliminary ablation.
- [Section 4] For reproducibility, please provide the exact query prompts, the Enron subset construction, and the model configuration used by each system (including which models the Palimpzest optimizer selected). Currently 'we recreated a document processing task from [15]' is not enough to reproduce the experiments.
- [Section 2.2 and Figure 2] The Context class's index method and user-defined tools are not exercised in the evaluation, so their claimed benefit is asserted rather than demonstrated. Also, there are minor typos (e.g., 'SmolAgent's CodeAgent' and inconsistent capitalization of search/compute operators).
Circularity Check
No significant circularity; all central claims are empirical comparisons against external baselines.
full rationale
The paper is a systems/vision paper with no formal derivation chain. Its central claims are empirical: (1) PZ compute achieves lower percentage error than a handcrafted semantic operator program and a naive CodeAgent on one query, and (2) PZ compute achieves higher F1 than a CodeAgent and comparable quality to CodeAgent+ while saving 76.8% cost and 72.7% runtime on another query. These are measured outcomes, not consequences derived from definitions. The CodeAgent+ baseline is described as using unoptimized semantic operators, which is a potential threat to the fairness of the comparison, but that is an experimental design issue, not circularity: the paper does not define CodeAgent+'s cost in terms of the prototype's output, nor does it fit any parameter to the reported savings. Self-citations to Kramabench, Palimpzest, and Abacus are used as background and as benchmark/task references; they are not load-bearing justifications for the prototype's performance, and the Enron task was published in prior work with its own ground truth. No equations are present, no fitted parameter is renamed as a prediction, and no uniqueness theorem or ansatz is smuggled in via self-citation. The paper's conclusion that optimized execution helps is a tautology in the sense that an optimizer is designed to improve cost/latency, but the magnitude of the improvement is empirically measured against an independently implemented baseline and is not implied by construction. Therefore, no circular step can be quoted or exhibited.
Axiom & Free-Parameter Ledger
axioms (4)
- domain assumption GPT-4o behavior is stable and representative for all systems and semantic operators.
- domain assumption Ground truth for the Kramabench query is fully contained in a single CSV file.
- ad hoc to paper The handcrafted semantic operator program and the CodeAgent baselines are representative implementations of their respective paradigms.
- domain assumption Semantic operators maintain iterator semantics that process one record at a time.
invented entities (4)
-
compute operator
independent evidence
-
search operator
independent evidence
-
Context abstraction
independent evidence
-
ContextManager
no independent evidence
Cite this review
Pith. "Pith review of Deep Research is the New Analytics System: Towards Building the Runtime for AI-Driven Analytics." pith.science (2026). https://pith.science/paper/7BJQMEG6
@misc{pith2026250902751,
author = {Pith},
title = {Pith review of: Deep Research is the New Analytics System: Towards Building the Runtime for AI-Driven Analytics},
year = {2026},
howpublished = {\url{https://pith.science/paper/7BJQMEG6}},
note = {Machine review of arXiv:2509.02751}
}
read the original abstract
With advances in large language models (LLMs), researchers are creating new systems that can perform AI-driven analytics over large unstructured datasets. Recent work has explored executing such analytics queries using semantic operators -- a declarative set of AI-powered data transformations with natural language specifications. However, even when optimized, these operators can be expensive to execute on millions of records and their iterator execution semantics make them ill-suited for interactive data analytics tasks. In another line of work, Deep Research systems have demonstrated an ability to answer natural language question(s) over large datasets. These systems use one or more LLM agent(s) to plan their execution, process the dataset(s), and iteratively refine their answer. However, these systems do not explicitly optimize their query plans which can lead to poor plan execution. In order for AI-driven analytics to excel, we need a runtime which combines the optimized execution of semantic operators with the flexibility and more dynamic execution of Deep Research systems. As a first step towards this vision, we build a prototype which enables Deep Research agents to write and execute optimized semantic operator programs. We evaluate our prototype and demonstrate that it can outperform a handcrafted semantic operator program and open Deep Research systems on two basic queries. Compared to a standard open Deep Research agent, our prototype achieves up to 1.95x better F1-score. Furthermore, even if we give the agent access to semantic operators as tools, our prototype still achieves cost and runtime savings of up to 76.8% and 72.7% thanks to its optimized execution.
Figures
Forward citations
Cited by 2 Pith papers
-
Kalypso: Relational LLM Serving
Query-aware LLM serving with adaptive memory-aware pipelining reuses cross-operator KV-cache state and speeds semantic queries by up to 4.57×.
-
Agent-Aided Design for Dynamic CAD Models
AADvark extends agent-aided CAD design to dynamic 3D assemblies with movable parts by integrating constraint solvers and visual feedback to create a verification signal for the agent.
Reference graph
Works this paper leans on
-
[1]
Shah, Benjamin Sowell, Dan Tecuci, Vinayak Thapliyal, and Matt Welsh
Eric Anderson, Jonathan Fritz, Austin Lee, Bohou Li, Mark Lindblad, Henry Lindeman, Alex Meyer, Parth Parmar, Tanvi Ranade, Mehul A. Shah, Benjamin Sowell, Dan Tecuci, Vinayak Thapliyal, and Matt Welsh. 2025. The Design of an LLM-powered Unstructured Analytics System. CIDR
work page 2025
-
[2]
Michael R. Anderson, Michael J. Cafarella, Germán Ros, and Thomas F. Wenisch
-
[3]
Anthropic. 2025. Claude takes research to new places . Retrieved July 31, 2025 from https://www.anthropic.com/news/research
work page 2025
-
[4]
Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ram- chandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. 2025. Why Do Multi-Agent LLM Systems Fail? arXiv:2503.13657 [cs.AI] https://arxiv.org/abs/ 2503.13657
Pith/arXiv arXiv 2025
-
[5]
S. Chaudhuri, R. Krishnamurthy, S. Potamianos, and K. Shim. 1995. Optimizing queries with materialized views. In Proceedings of the Eleventh International Conference on Data Engineering . 190–200. doi:10.1109/ICDE.1995.380392
-
[6]
William W. Cohen. 2015. Enron Email Dataset. https://www.cs.cmu.edu/~enron/
work page 2015
-
[7]
Benoit Dageville, Thierry Cruanes, Marcin Zukowski, Vadim Antonov, Artin Avanes, Jon Bock, Jonathan Claybaugh, Daniel Engovatov, Martin Hentschel, Jiansheng Huang, Allison W. Lee, Ashish Motivala, Abdul Q. Munir, Steven Pelley, Peter Povinec, Greg Rahn, Spyridon Triantafyllis, and Philipp Unterbrunner. 2016. The Snowflake Elastic Data Warehouse. In Procee...
-
[8]
Google. 2025. Gemini Deep Research. Retrieved July 31, 2025 from https://gemini. google/overview/deep-research/
work page 2025
-
[9]
Anurag Gupta, Deepak Agarwal, Derek Tan, Jakub Kulesza, Rahul Pathak, Ste- fano Stefani, and Vidhya Srinivasan. 2015. Amazon Redshift and the Case for Simpler Data Warehouses. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data (Melbourne, Victoria, Australia) (SIGMOD ’15). Association for Computing Machinery, New York, NY...
- [10]
-
[11]
Daniel Kang, John Emmons, Firas Abuzaid, Peter Bailis, and Matei Zaharia. 2017. NoScope: optimizing neural network queries over video at scale. Proc. VLDB Endow. 10, 11 (Aug. 2017), 1586–1597. doi:10.14778/3137628.3137664
arXiv 2017
-
[12]
Daniel Kang, John Guibas, Peter Bailis, Tatsunori Hashimoto, Yi Sun, and Matei Zaharia. 2021. Accelerating approximate aggregation queries with expensive predicates. Proc. VLDB Endow.14, 11 (July 2021), 2341–2354. doi:10.14778/3476249. 3476285
-
[13]
Eugenie Lai, Gerardo Vitagliano, Ziyu Zhang, Sivaprasad Sudhir, Om Chabra, Anna Zeng, Anton A. Zabreyko, Chenning Li, Ferdi Kossmann, Jialin Ding, Jun Chen, Markos Markakis, Matthew Russo, Weiyang Wang, Ziniu Wu, Michael J. Cafarella, Lei Cao, Samuel Madden, and Tim Kraska. 2025. KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Dat...
arXiv 2025
-
[14]
LangChain. 2025. Open Deep Research . Retrieved July 31, 2025 from https: //blog.langchain.com/open-deep-research/
work page 2025
-
[15]
Chunwei Liu, Matthew Russo, Michael Cafarella, Lei Cao, Peter Baile Chen, Zui Chen, Michael Franklin, Tim Kraska, Samuel Madden, Rana Shahout, et al. 2025. Palimpzest: Optimizing AI-Powered Analytics with Declarative Query Processing. CIDR
2025
-
[16]
Duo Lu, Siming Feng, Jonathan Zhou, Franco Solleza, Malte Schwarzkopf, and Uğur Çetintemel. 2025. VectraFlow: Integrating Vectors into Stream Processing. CIDR
work page 2025
-
[17]
OpenAI. 2025. Deep Research System Card . Retrieved July 31, 2025 from https: //cdn.openai.com/deep-research-system-card.pdf
work page 2025
-
[18]
Liana Patel, Siddharth Jha, Melissa Pan, Harshit Gupta, Parth Asawa, Carlos Guestrin, and Matei Zaharia. 2025. Semantic Operators: A Declarative Model for Rich, AI-based Data Processing. arXiv:2407.11418 [cs.DB] https://arxiv.org/abs/ 2407.11418
Pith/arXiv arXiv 2025
-
[19]
Perplexity. 2025. Introducing Perplexity Deep Research . Retrieved July 31, 2025 from https://www.perplexity.ai/hub/blog/introducing-perplexity-deep-research
work page 2025
-
[20]
Mark Raasveldt and Hannes Mühleisen. 2019. DuckDB: an Embeddable Analytical Database. In Proceedings of the 2019 International Conference on Management of Data (Amsterdam, Netherlands) (SIGMOD ’19). Association for Computing Machinery, New York, NY, USA, 1981–1984. doi:10.1145/3299869.3320212
-
[21]
Matthew Russo, Tatsunori Hashimoto, Daniel Kang, Yi Sun, and Matei Zaharia
-
[22]
Matthew Russo, Sivaprasad Sudhir, Gerardo Vitagliano, Chunwei Liu, Tim Kraska, Samuel Madden, and Michael Cafarella. 2025. Abacus: A Cost-Based Optimizer for Semantic Operator Systems. arXiv:2505.14661 [cs.DB] https://arxiv.org/abs/ 2505.14661
arXiv 2025
-
[23]
Dario Satriani, Enzo Veltri, Donatello Santoro, Sara Rosato, Simone Varriale, and Paolo Papotti. 2025. Logical and Physical Optimizations for SQL Query Execution over Large Language Models(SIGMOD ’25). Association for Computing Machinery, New York, NY, USA, 28 pages. doi:10.1145/3725411
doi:10.1145/3725411 2025
-
[24]
Shreya Shankar, Tristan Chambers, Tarak Shah, Aditya G. Parameswaran, and Eugene Wu. 2024. DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing. arXiv:2410.12189 [cs.DB] https://arxiv.org/abs/2410.12189
Pith/arXiv arXiv 2024
-
[25]
SmolAgents. 2024. Introducing smolagents, a simple library to build agents . Re- trieved August 1, 2025 from https://huggingface.co/blog/smolagents
work page 2024
-
[26]
SmolAgents. 2025. Open-source DeepResearch – Freeing our search agents . Re- trieved July 31, 2025 from https://huggingface.co/blog/open-deep-research
work page 2025
-
[27]
Mike Stonebraker, Daniel J. Abadi, Adam Batkin, Xuedong Chen, Mitch Cherniack, Miguel Ferreira, Edmond Lau, Amerson Lin, Sam Madden, Elizabeth O’Neil, Pat O’Neil, Alex Rasin, Nga Tran, and Stan Zdonik. 2005. C-store: a column-oriented DBMS. In Proceedings of the 31st International Conference on Very Large Data Bases (Trondheim, Norway) (VLDB ’05). VLDB En...
work page 2005
-
[28]
Matthias Urban and Carsten Binnig. 2023. CAESURA: Language Models as Multi- Modal Query Planners. arXiv:2308.03424 [cs.DB] https://arxiv.org/abs/2308. 03424
work page internal anchor Pith review Pith/arXiv arXiv 2023
-
[29]
xAI. 2025. Grok 3 Beta — The Age of Reasoning Agents . Retrieved July 31, 2025 from https://x.ai/news/grok-3
work page 2025
-
[30]
Gonzalez, Clark Barrett, and Ying Sheng
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Lan- guage Model Programs. arXiv:2312.07104 [cs.AI] https://arxiv.org/abs/2312.07104
Pith/arXiv arXiv 2024
-
[2018]
2019 IEEE 35th International Conference on Data Engineering (ICDE) (2018), 1466–1477
Physical Representation-Based Predicate Optimization for a Visual Analyt- ics Database. 2019 IEEE 35th International Conference on Data Engineering (ICDE) (2018), 1466–1477. https://api.semanticscholar.org/CorpusID:48362547
work page 2019
- [2023]
This paper was first reviewed by deepseek-v4-flash on August 5, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.