Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Data Agent: A Holistic Architecture for Orchestrating Data+AI Ecosystems

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proposes that an LLM-centered 'Data Agent' architecture can autonomously understand data and queries, orchestrate and optimize pipelines, and self-reflect on results, replacing human orchestration across data science…

desk verdict A solid vision paper with a genuinely new iDataScience design, but the agent-selection loop is closed—LLM-generated benchmarks and embeddings may rank agents by agreement with the generator rather than by competence. read the letter →

arxiv 2507.01599 v1 pith:SWMEWM5D submitted 2025-07-02 cs.DB cs.AIcs.CLcs.LG

Zhaoyan Sun , Jiayi Wang , Xinyang Zhao , Jiachi Wang , Guoliang Li This is my paper
classification cs.DBcs.AIcs.CLcs.LG
keywords DataAgentlargelanguagemodelsData+AImulti-agentsystemspipelineorchestrationskillbenchmarksemanticoperatorsDBA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes that large language models can close the semantic-understanding, reasoning, and planning gaps that keep today's data systems from adapting to new data, queries, and environments. It introduces the 'Data Agent,' an architecture that combines perception, reasoning and planning, tool invocation, memory, continuous learning, and multi-agent coordination so that data-related tasks can be carried out with minimal human orchestration. The centerpiece is iDataScience, a multi-agent system that builds a skill-based benchmark offline, selects agents through fine-tuned task embeddings and adaptive score aggregation, and orchestrates pipelines online with dynamic refinement. The paper also sketches analytics agents for unstructured, semantic structured, data-lake, and multi-modal data, plus an LLM-powered DBA agent. What would matter if this works: the same orchestration machinery could span data science, analytics, and database diagnosis, replacing hand-tuned pipelines in each domain.

What carries the argument

The load-bearing object is the 'Data Agent' itself: a unified wrapper that couples six capabilities—perception, reasoning and planning, tool invocation, memory, continuous learning, and multi-agent collaboration—around the data-processing stack. Inside iDataScience, the mechanism that carries the argument is the skill-based benchmark plus fine-tuned task embedding pair: data skills, meaning the fundamental data science capabilities such as column derivation and linear regression, are extracted from a corpus, clustered into a hierarchy, weighted by importance, and recombined by an LLM into test cases with executable evaluation functions; a contrastively fine-tuned embedding model then maps an online task near test cases that demand the same skills, and agent scores are aggregated by similarity to those neighbors. That machinery converts 'pick the right agent for this task' into a similarity-weighted ranking problem, and it feeds a planner that decomposes tasks, assigns agents, merges or splits sub-tasks, and re-plans when intermediate results fail.

What would settle it

A concrete falsifier: build two non-overlapping data science corpora, generate benchmarks from each with different LLM generators, and run the same set of candidate agents; if the agent selected by adaptive benchmark aggregation changes with the generator even when human-graded outcomes of the agents are unchanged, the selection mechanism is tracking generator bias rather than competence.

Watch

Extended reading notes

Core claim

The paper's central claim is that end-to-end data work—data science, analytics, and database diagnosis—can be delegated to an LLM-centered agent that understands its data and tools, plans multi-step pipelines, executes them, and revises them from intermediate results. To make this concrete, iDataScience decomposes an online task into sub-tasks, assigns each to the best agent using a benchmark built from hierarchically organized 'data skills,' and refines the pipeline at both the agent level and the global level. The key selection mechanism aligns task embeddings with the text embeddings of correct solutions through contrastive learning, so that similar tasks get similar capability requirements. The data analytics agents operationalize the same idea through semantic operators that match natural-language segments to logical plans and then optimize physical execution plans. The DBA agent is described as extracting knowledge from diagnostic documents and using tree search to identify root causes of database anomalies, with reported results beating GPT-4 on previously unseen anomalies.

Load-bearing premise

The approach rests on assuming that the LLM-generated benchmark and the embeddings built from LLM-written solutions rank agents by genuine competence rather than by similarity to the generator's preferences—an assumption the paper leaves untested (Section 6 instead lists theoretical guarantees as open).

Editorial extensions

If this is right

  • If the architecture works, the same orchestration layer could serve many data tasks: analytics, data science, data lake querying, and DBA work would share one semantic catalog, agent-selection, and pipeline-refinement machinery rather than bespoke per-task pipelines.
  • Benchmarks would no longer be fixed lists of pre-defined task types; they could be composed on demand from an expandable skill hierarchy, so evaluation can track new capabilities as agents are added.
  • Agent selection would improve with usage: newly integrated agents can start from document analysis and move to full benchmark execution once resources allow, making the system extensible.
  • Dynamic pipeline refinement—rephrasing sub-tasks, choosing the next-best agent, or re-planning globally with intermediate results stored in a data catalog—would make failures less catastrophic and reduce redundant computation.
  • If the DBA claim generalizes, database diagnosis could shift from manual rule and monitoring pipelines to LLM agents that keep knowledge fresh from documents and diagnose unseen anomalies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the benchmark's test cases and evaluation functions are generated by LLMs, and the candidate agents are themselves LLM-based, the ranking could partly reflect stylistic similarity to the generator rather than task competence; a cross-generator evaluation would separate the two effects.
  • The same skill-hierarchy-plus-embedding selection procedure could be turned into a general 'agent router' for other Data+AI chores such as index tuning, query rewrite, or data cleaning, where the paper's architecture is sketched but not detailed.
  • A testable prediction follows from the embedding design: tasks with similar solution procedures should yield similar agent rankings even when their domains and data formats differ; this is what the contrastive alignment is meant to guarantee and can be checked directly.
  • The paper leaves theoretical guarantees open; a natural extension would be to bound the degradation introduced by LLM hallucinations in semantic operators, for instance by adding verifier checks at pipeline checkpoints.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes 'Data Agent,' a conceptual architecture for integrating LLM-based reasoning, planning, and tool invocation into Data+AI systems. It describes a multi-agent system called iDataScience, which includes offline benchmark construction from LLM-discovered data skills and online pipeline orchestration with adaptive agent selection (Sections 3.2–3.4). The paper also surveys data analytics agents and a DBA agent, and lists open challenges. The central claim is that such an architecture can autonomously handle data-related tasks end-to-end. No experiments, benchmarks, ablation studies, or code are presented; the only reported empirical claim ('Our results demonstrate...' in Section 5) refers to prior D-Bot publications rather than to this work.

Significance. If the iDataScience pipeline were validated, the paper would provide a useful organizing framework for LLM-based data agent orchestration. The proposed decomposition into benchmarking, adaptive selection, and pipeline refinement is plausible and aligns with current trends. However, the significance is severely constrained by the absence of any empirical evaluation and by the self-referential construction of the benchmark used for agent selection. The authors explicitly note in Section 6 that theoretical guarantees remain open, so the paper currently offers a vision rather than an established method. The existing D-Bot work is likely valuable, but it is not evidence for the claims made about iDataScience.

major comments (3)
  1. [§3.2–3.3] The core adaptive benchmark aggregation loop is internally circular. LLM S filters examples, extracts data skills (Section 3.2.1), synthesizes test cases, and writes executable evaluation functions (Section 3.2.2, including cases where 'LLM S can be invoked within the evaluation function'). The task embedding model is then fine-tuned to align task representations with the pre-trained text embeddings of those same LLM-generated solutions (Section 3.3.1). Finally, Section 3.3.2 weights test cases by embedding similarity and calls the resulting score 'unbiased.' If candidate agents share inductive biases with LLM S, or if evaluation functions reward mere mention of required skills, the rankings produced by this procedure may reflect agreement with the generator rather than competence on arbitrary tasks. The paper provides no external validation (e.g., against DSBench or DABench, which are cited but never used) to break this circularity. Since adaptive benchmark aggregation is the load-bearing mechanism for the claimed 'adaptive' and 'unbiased' agent selection, this concern directly affects the central claim. The authors should either provide held-out validation comparing the selection method's predictions against independent task performance, or substantially temper the claims about unbiasedness and adaptivity.
  2. [§5] The sentence 'Our results demonstrate that the DBA agent significantly outperforms traditional methods and standard models like GPT-4' is unsupported by anything in this paper. The cited references [42, 43] are prior D-Bot publications, and the current manuscript contains no experimental setup, dataset, metrics, or analysis for the DBA agent. This is not a presentation issue; it is an unsupported empirical claim presented as a result of this work. Either the qualifying evidence should be presented, or the sentence should be rewritten to clearly attribute the results to earlier D-Bot papers.
  3. [§3 (overall)] The manuscript presents iDataScience as a concrete system with algorithmic components (hierarchical skill discovery, benchmark construction, task embedding fine-tuning, agent selection, pipeline refinement), but it provides no experimental evaluation of any of these components. There are no benchmarks run, no comparisons to baselines, no ablations of the components (e.g., the effect of the fine-tuned embedding vs. a generic embedding, or the choice of k and top-k), and no released code or data. Table 1 is a qualitative comparison with no measured values. As a result, the central claim that iDataScience 'adaptively handle[s] data science tasks' is an unverified assertion. At minimum, the paper should clearly state that these are proposed mechanisms without empirical validation, and should not use phrases such as 'our results' for prior work.
minor comments (5)
  1. [§2] The sentence 'Semantic data organization and semantic indexes are also very improve to improve the data agent efficiency' contains a grammatical error ('are also very improve'); it should likely read 'also very important for improving.' Please revise.
  2. [§3.2.1] The threshold for leaf-node clusters and the value of k in sampled skills are introduced as user-specified or pre-defined parameters, but no guidance or sensitivity analysis is given. Since these parameters directly affect benchmark coverage and complexity, their choice may materially alter the results; a brief discussion of reasonable ranges would improve reproducibility.
  3. [§3.3.2] The phrase 'unbiased evaluation score' is used without defining what 'unbiased' means in this context. The text explains that weights reflect embedding similarity, but this does not in itself establish unbiasedness. Please either provide a formal definition or replace 'unbiased' with a more neutral term such as 'similarity-weighted.'
  4. [§4] The descriptions of the three data analytics agents are high-level and largely repeat the same architectural components (semantic operators, logical/physical plans, adaptive execution). A comparative table or a single paragraph highlighting the specific differences among unstructured, semantic structured, data-lake, and multi-modal agents would make the section more informative.
  5. [§6] The 'Data Agent Benchmark' challenge acknowledges that developing benchmarks is an open problem, which is somewhat in tension with the detailed benchmark construction method in Section 3.2. Please clarify the distinction between the proposed skill-based benchmark and the community-wide benchmarks that the paper identifies as an open challenge.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper proposes an architecture without a derivation chain whose outputs are forced by its inputs.

full rationale

The paper is a systems/architecture position paper: it proposes the Data Agent concept and sketches iDataScience rather than deriving a quantitative result, so there is no derivation chain whose conclusion could be equivalent to its premises. The only mechanism that might look self-referential is the offline/online loop in Sections 3.2–3.3: LLM S extracts skills, synthesizes test cases, writes evaluation functions, and is also used to summarize solutions that guide the fine-tuned task embedding, after which candidate agents are ranked by aggregated benchmark scores. However, the ranking is computed by actually executing candidate agents on those test cases and aggregating the observed evaluation scores; nothing in the paper's equations makes the ranking algebraically equal to the LLM generator's preferences. The similarity-weighted aggregation in Section 3.3.2 is a stated design choice, not a tautology, and the paper even acknowledges its limitations by listing theoretical guarantees as an open challenge (Section 6). The DBA agent performance claim relies on the authors' prior work [42,43], but those are externally evaluable experimental systems, and the claim is illustrative rather than load-bearing for the paper's central architectural thesis. A shared-generator bias in the benchmark would be a validity threat or correctness risk, not a circular reduction, and the instructions require exhibiting a specific equation-level reduction before flagging circularity. None exists here, so the honest finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The central proposal rests on several unvalidated assumptions about LLM reliability and the validity of LLM-generated benchmarks, plus a handful of unspecified design hyperparameters. The paper itself flags theoretical guarantee and self-reflection as open challenges (Section 6), acknowledging the fragility of these assumptions.

free parameters (4)
  • k (number of sampled data skills per test case) = null (user-specified)
    Section 3.2.2: 'we sample a random set of k data skills from the leaf nodes...' and later 'users can specify to tailor the benchmark.' This is a hand-chosen hyperparameter affecting test case complexity.
  • leaf-node cluster size threshold = null (pre-defined but unspecified)
    Section 3.2.1: 'If the skill number within some cluster falls below a pre-defined threshold, such clusters are designated as leaf nodes.' The threshold controls hierarchy granularity and is not specified.
  • top-k relevant test cases for adaptive aggregation = null
    Section 3.3.2: 'identify the most relevant test cases with top-k embedding similarities.' k is a retrieval hyperparameter; the paper gives no value or heuristic.
  • agent selection score threshold for sub-task decomposition = null
    Section 3.4.1: 'if no suitable data agent can be selected for a given sub-task (e.g., all agents with evaluation scores below a pre-defined threshold).' The threshold is undefined and affects planning behavior.
assumptions (5)
  • domain assumption LLMs possess sufficient semantic understanding, reasoning, and planning capabilities to orchestrate complex data pipelines.
    Stated in the abstract and Section 1 as the motivation for Data Agents ('Fortunately, we have witnessed the success of large language models (LLMs) in enhancing semantic understanding, reasoning, and planning abilities.'). No empirical evidence is given in this paper.
  • domain assumption Data skills extracted by LLM from example solutions accurately represent the capabilities required for data science tasks.
    Section 3.2.1: 'we instruct it to summarize the procedure... use LLM S to extract associated data skills.' This is the foundation of the benchmark; unvalidated.
  • ad hoc to paper The data-skill-based benchmark, with evaluation functions written by LLM, provides an unbiased measure of agent performance.
    Section 3.2.2 constructs test cases and evaluation functions via LLM; Section 3.3 uses them for agent selection. This is a central ad hoc assumption with no external validation.
  • domain assumption Fine-tuned task embeddings, trained on LLM-generated solution snippets, capture task capability requirements and generalize to new tasks.
    Section 3.3.1: 'We employ a contrastive learning framework... Multiple Negatives Ranking Loss.' The alignment's transferability to unseen tasks is assumed.
  • domain assumption Agent profiles constructed from documents and benchmark deviations predict online agent performance on new sub-tasks.
    Section 3.4.1: data agent profile construction uses 'design principle', 'representative example', 'experiment result'; selection relies on these profiles. No evidence provided.
invented entities (1)
  • Data Agent
    purpose: Autonomous orchestrator of Data+AI tasks via LLM-driven understanding, planning, tool invocation, and reflection.
    The paper introduces 'Data Agent' as a new conceptual architecture (Section 2). It is not a physical entity, and no experimental or external evidence supports its effectiveness; the paper presents it as a proposal.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Data Agent: A Holistic Architecture for Orchestrating Data+AI Ecosystems." pith.science (2026). https://pith.science/paper/SWMEWM5D

@misc{pith2026250701599,
  author       = {Pith},
  title        = {Pith review of: Data Agent: A Holistic Architecture for Orchestrating Data+AI Ecosystems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SWMEWM5D}},
  note         = {Machine review of arXiv:2507.01599}
}
read the original abstract

Traditional Data+AI systems utilize data-driven techniques to optimize performance, but they rely heavily on human experts to orchestrate system pipelines, enabling them to adapt to changes in data, queries, tasks, and environments. For instance, while there are numerous data science tools available, developing a pipeline planning system to coordinate these tools remains challenging. This difficulty arises because existing Data+AI systems have limited capabilities in semantic understanding, reasoning, and planning. Fortunately, we have witnessed the success of large language models (LLMs) in enhancing semantic understanding, reasoning, and planning abilities. It is crucial to incorporate LLM techniques to revolutionize data systems for orchestrating Data+AI applications effectively. To achieve this, we propose the concept of a 'Data Agent' - a comprehensive architecture designed to orchestrate Data+AI ecosystems, which focuses on tackling data-related tasks by integrating knowledge comprehension, reasoning, and planning capabilities. We delve into the challenges involved in designing data agents, such as understanding data/queries/environments/tools, orchestrating pipelines/workflows, optimizing and executing pipelines, and fostering pipeline self-reflection. Furthermore, we present examples of data agent systems, including a data science agent, data analytics agents (such as unstructured data analytics agent, semantic structured data analytics agent, data lake analytics agent, and multi-modal data analytics agent), and a database administrator (DBA) agent. We also outline several open challenges associated with designing data agent systems.

Figures

Figures reproduced from arXiv: 2507.01599 by the authors.

Figure 1
Figure 1. Challenges of Data+AI Systems. heavily on experts tuning to accommodate different scenarios. On the other hand, for Data4AI, our community extends database optimization techniques to ease the deployment of AI, including in-database machine learning (ML) training and inference [16], data preparation [17], data cleaning [18], data integration [19], feature man￾agement [20], and model management [21]. The main obstacle… view at source ↗
Figure 2
Figure 2. Key Factors of Data Agents. 2 Data Agent The Data Agent is designed to autonomously carry out data-related tasks with capabilities for knowledge com￾prehension, automatic planning, and self-reflection. Data Agents require to consider six key factors as shown in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Data Agents Framework. metadata index), enhancing data access performance. The data fabric provides a unified view of heterogeneous data by linking and integrating diverse data, allowing easy data retrieval by the agent. Semantic data organization and semantic indexes are also very improve to improve the data agent efficiency. Importantly, there are numerous tools for data preparation, cleaning, and integration. Thi… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Data Agents Architecture. protocols, we can facilitate communication between agents and synchronize their statuses via A2A [25]. Agent Execution aims to execute multiple agents either in a pipeline or in parallel, enhancing the system’s fault toler￾ance and fast recove…
Figure 5
Figure 5. Figure 5: Overview of iDataScience. Besides, to ensure unbiased agent evaluation for an online task, benchmark test cases should be adaptively aggregated based on their similarity to the task. We thus construct an efficient index to enhance the performance of similarity search o…
Figure 6
Figure 6. Figure 6: Example of Data Skill-based Benchmark Construction. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Example of Fine-tuned Data Science Task Embedding. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. AgenticDataBench: A Comprehensive Benchmark for Data Agents

    cs.DB 2026-07 unverdicted novelty 5.0 of 10

    AgenticDataBench is a new benchmark covering realistic data science tasks across 15 domains using extracted skills and LLM-generated workflows to evaluate data agents at fine granularity.

  2. BetaWeb: Towards a Blockchain-enabled Trustworthy Agentic Web

    cs.MA 2025-08 unverdicted novelty 4.0 of 10

    BetaWeb promises a blockchain-enabled trustworthy agentic web, but the submitted manuscript body is a different mining-robot paper, leaving the proposal without supporting evidence.

Reference graph

Works this paper leans on

44 extracted references · 38 canonical work pages · cited by 2 Pith papers

  1. [1]

    G. Li, X. Zhou, L. Cao. AI meets database: AI4DB and DB4AI. SIGMOD, 2021

    work page 2021

  2. [2]

    Y . Wu, X. Zhou, Y . Zhang, G. Li. Automatic Database Index Tuning: A Survey.TKDE, 2024

    work page 2024

  3. [3]

    W. Zhou, C. Lin, X. Zhou, G. Li. Breaking It Down: An In-Depth Study of Index Advisors. VLDB, 2024

    work page 2024

  4. [4]

    Y . Han, C. Chai, J. Liu, G. Li, C. Wei, et al. Dynamic materialized view management using graph neural network. ICDE, 2023

    work page 2023

  5. [5]

    X. Zhou, G. Li, J. Feng, L. Liu, W. Guo. Grep: A graph learning based database partitioning system. SIGMOD, 2023

    work page 2023

  6. [6]

    X. Zhao, X. Zhou, G. Li. Automatic database knob tuning: A survey. TKDE, 2023

    work page 2023

  7. [7]

    Marcus, P

    R. Marcus, P. Negi, H. Mao, N. Tatbul, M. Alizadeh, et al. Bao: Making learned query optimization practical. SIGMOD, 2021

    work page 2021

  8. [8]

    X. Zhou, G. Li, C. Chai, J. Feng. A learned query rewrite system using monte carlo tree search. VLDB, 2021

    work page 2021

Show all 44 references
  1. [9]

    X. Yu, C. Chai, G. Li, J. Liu. Cost-based or learning-based? A hybrid query optimizer for query plan selection. VLDB, 2022

    2022

  2. [10]

    J. Wang, C. Chai, J. Liu, G. Li. FACE: A normalizing flow based cardinality estimator. VLDB, 2021

    2021

  3. [11]

    J. Sun, J. Zhang, Z. Sun, G. Li, N. Tang. Learned cardinality estimation: A design space exploration and a comparative evaluation. VLDB, 2021

    2021

  4. [12]

    Hilprecht and C

    B. Hilprecht and C. Binnig. Zero-Shot Cost Models for Out-of-the-box Learned Cost Prediction. VLDB, 2022

    2022

  5. [13]

    Wehrstein, C

    J. Wehrstein, C. Binnig, F. ¨Ozcan, S. Vasudevan, Y . Gan, et al. Towards Foundation Database Models.CIDR, 2025

    2025

  6. [14]

    X. Zhou, J. Sun, G. Li, J. Feng. Query performance prediction for concurrent queries using graph embedding. VLDB, 2020

    2020

  7. [15]

    Z. Sun, X. Zhou, G. Li. Learned index: A comprehensive experimental evaluation. VLDB, 2023

    2023

  8. [16]

    Y . Guo, G. Li, R. Hu, Y . Wang. In-database query optimization on SQL with ML predicates.The VLDB Journal, 34 (1), 12, 2025

    2025

  9. [17]

    C. Chai, J. Liu, N. Tang, J. Fan, D. Miao, et al. In-database query optimization on SQL with ML predicates.SIGMOD, 2023

    2023

  10. [18]

    Siddiqi, R

    S. Siddiqi, R. Kern, M. Boehm. SAGA: A Scalable Framework for Optimizing Data Cleaning Pipelines for Machine Learning Applications. SIGMOD, 2023

    2023

  11. [19]

    J. Wang, G. Li. Aop: Automated and interactive llm pipeline orchestration for answering complex queries. CIDR, 2025

    2025

  12. [20]

    X. Zhou, C. Chen, K. Li, B. He, M. Lu, et al. Febench: A benchmark for real-time relational data feature extraction. VLDB, 2023. 15

    2023

  13. [21]

    Moritz, R

    P. Moritz, R. Nishihara, S. Wang, A. Tumanov, R. Liaw, et al. Ray: A Distributed Framework for Emerging AI Applications. OSDI, 2018

    2018

  14. [22]

    Hurst, A

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, et al. GPT-4o System Card. arXiv:2410.21276, 2024

    2024 arXiv

  15. [23]

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv:2501.12948, 2025

    2025 arXiv

  16. [24]

    Introducing the Model Context Protocol

    Anthropic. Introducing the Model Context Protocol. Newsroom Anthropic , 2024

    2024

  17. [25]

    Surapaneni, M

    R. Surapaneni, M. Jha, M. Vakoc, T. Segal. Announcing the Agent2Agent Protocol (A2A). Google for Developers, 2025

    2025

  18. [26]

    X. Hu, Z. Zhao, S. Wei, Z. Chai, Q. Ma, et al. InfiAgent-DABench: evaluating agents on data analysis tasks. ICML, 2024

    2024

  19. [27]

    L. Jing, Z. Huang, X. Wang, W. Yao, W. Yu, et al. DSBench: How Far Are Data Science Agents to Becoming Data Science Experts?. ICLR, 2025

    2025

  20. [28]

    Didolkar, A

    A. Didolkar, A. Goyal, N. R. Ke, S. Guo, M. Valko, T. Lillicrap, et al. Metacognitive capabilities of llms: An exploration in mathematical problem solving. NIPS, 2024

    2024

  21. [29]

    Sarthi, S

    P. Sarthi, S. Abdullah, A. Tuli, S. Khanna, A. Goldie, et al. RAPTOR: Recursive Abstractive Processing for Tree- Organized Retrieval. ICLR, 2024

    2024

  22. [30]

    McInnes, J

    L. McInnes, J. Healy, J. Melville. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduc- tion. arXiv:1802.03426, 2020

    2020 arXiv

  23. [31]

    Z. Liu, P. Wang, R. Xu, S. Ma, C. Ruan, et al. Inference-time scaling for generalist reward modeling. arXiv:2504.02495, 2025

    2025

  24. [32]

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, et al. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12, 2024

    2024

  25. [33]

    D. Yu, S. Kaur, A. Gupta, J. Brown-Cohen, A. Goyal, et al. Skill-Mix: a Flexible and Expandable Family of Evaluations for AI models. ICLR, 2024

    2024

  26. [34]

    Jiang, R

    Z. Jiang, R. Meng, X. Yang, S. Yavuz, Y . Zhou, et al. Vlm2vec: Training vision-language models for massive multimodal embedding tasks. ICLR, 2025

    2025

  27. [35]

    Reimers, I

    N. Reimers, I. Gurevych. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. EMNLP-IJCNLP, 2019

    2019

  28. [36]

    J. Fine, M. Kolla, I. Soloducho. Data Science Agent in Colab: The future of data analysis with Gemini. Google for Developers, 2025

    2025

  29. [37]

    S. Hong, Y . Lin, B. Liu., B. Liu, B. Wu, et al. Data Interpreter: An LLM Agent For Data Science.arXiv:2402.18679, 2024

    2024 arXiv

  30. [38]

    Z. You, Y . Zhang, D. Xu, Y . Lou, Y . Yan, et al. DatawiseAgent: A Notebook-Centric LLM Agent Framework for Automated Data Science. arXiv:2503.07044, 2025

    2025

  31. [39]

    Trirat, W

    P. Trirat, W. Jeong, Wonyong, S. J. Hwang. AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML. ICML, 2025

    2025

  32. [40]

    A. Li,Y . Xie, S. Li,F. Tsung, B. Ding, et al. Agent-Oriented Planning in Multi-Agent Systems. ICLR, 2025

    2025

  33. [41]

    J. Wang, G. Li, J. Feng. iDataLake: An LLM-Powered Analytics System on Data Lakes. IEEE Data Eng. Bull. , 49(1), 57-69, 2025

    2025

  34. [42]

    X. Zhou, G. Li, Z. Sun, Z. Liu, W. Chen, et al. D-bot: Database diagnosis system using large language models. VLDB, 2024

    2024

  35. [43]

    Z. Sun, X. Zhou, J. Wu, W. Zhou, G. Li. D-Bot: An LLM-Powered DBA Copilot. SIGMOD-Companion, 2025

    2025

  36. [44]

    Huang, W

    L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2), 1-55, 2025. 16

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.