Pith. sign in

REVIEW 2 major objections 4 minor 171 references

This paper argues that optimizing multi-agent LLM pipelines is a fundamentally new query optimization problem, requiring joint choice of workflow topology, models, and execution engines under latency, cost, and accuracy objectives — and tha

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 17:16 UTC pith:7OYFN7MK

load-bearing objection The vision is timely and the workload analysis is a real contribution, but the abstract's headline numbers are nowhere in the body—conditional on reporting or withdrawing them. the 2 major comments →

arxiv 2512.11001 v2 pith:7OYFN7MK submitted 2025-12-10 cs.DB cs.MA

Rethinking Query Optimization for Multi-Agent Systems [Vision]

Zoi Kaoudi , Ioana Giurgiu This is my paper
classification cs.DB cs.MA
keywords multi-agent workflowsquery optimizationlarge language modelsworkflow topologymodel selectionengine selectionPareto optimizationsemantic caching
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper's central claim is that optimizing multi-agent LLM workflows is a fundamentally different query optimization problem, not a tweak of classic database optimization: the optimizer must jointly choose the workflow topology (chain, DAG, feedback graph), the model behind each stochastic agent, and the engine behind each deterministic task, while balancing latency, monetary cost, and accuracy. Supporting this, the paper analyzes over 9000 multi-agent workflows and reports a controlled experiment on a 10-agent customer-support pipeline in which execution cost varied 153x, latency 5x, and quality 25% across reasonable configurations. The authors see systematic inefficiency in how humans build these systems — simple chains dominate, a small set of familiar open LLM families is reused, and a handful of engines do most of the work — which is precisely the pattern a query optimizer exploits. If the claim holds, a framework such as the proposed NOMA, built around a multi-objective planner, unified learned cost models, and a semantic cache, would let users navigate explicit cost/latency/accuracy trade-offs and capture order-of-magnitude savings. This is a vision paper: it defines the problem, sketches the architecture, and lists open challenges rather than delivering a working optimizer.

Core claim

On the paper's own terms, the discovery is that multi-agent workflow optimization is a genuinely new regime for query optimization because the search space is three coupled layers — how agents are wired together, which model each stochastic agent uses, and which engine each deterministic task runs on — with objectives that include monetary cost and probabilistic accuracy alongside latency. No common operator algebra spans the agents and engines, so classical cost-based rewriting does not transfer directly. The paper's corpus analysis finds heavy concentration on simple chains, on a few familiar open LLM families, and on relational engines for the roughly 43% of tasks that are deterministic,

What carries the argument

The central object is the abstract-to-executable workflow mapping problem: an abstract workflow W = (A, E) names agent capabilities and dependencies without committing to models or engines, and the task is to find executable workflows W* that instantiate each agent with a model and an engine under a set of objectives. The machinery carrying the argument is the NOMA optimization loop — a multi-objective planner that enumerates candidate W* over a search space defined by an agent registry, a model pool, and an engine pool; a unified cost model that mixes learned estimators for latency, token usage, and accuracy with engine statistics; a multi-layer semantic cache with an exact-match tier and a

Load-bearing premise

The load-bearing premise is that the analyzed corpus of 9000+ workflows (generated from tens of concrete examples) and the abstract's 10-agent controlled experiment — which is asserted in the abstract but not reported in the body — are representative of real deployments; if the generator baked in the chain-heavy, model-repetitive patterns, or if the experiment's 153x/5x/25% figures don't reproduce, the claim that manual configurations are systematically inefficient loses its

What would settle it

Re-running the claimed controlled experiment on a 10-agent pipeline: if enumerating reasonable model, engine, and structure configurations yields cost variance far smaller than 153x (e.g., under 2x), or if a corpus of independently collected production agent workflows shows no systematic concentration of chain topologies, model families, and engine choices, the paper's central motivation weakens.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the optimization problem is as the paper describes, any system that hard-codes one topology, one model per task, and one engine is leaving predictable, quantifiable savings unclaimed; even a moderate optimizer should beat manual configurations by multiples on cost.
  • Jointly choosing model and engine per agent becomes necessary: selecting a smaller model to cut token cost may raise downstream latency or accuracy loss, so layer-by-layer independent selection is provably insufficient.
  • Semantic caching moves from optional to core: with most tasks shared across workflows, the paper's argument implies that exact-match and embedding-based reuse of intermediate results is a primary lever for both cost and latency, not an add-on.
  • Cost models must become distribution-aware: because LLM latency and accuracy are non-deterministic and even non-monotonic under batching, the paper implies that optimizers must reason about variance and risk, not just expected values.
  • A community-wide research agenda follows: the paper claims that new search, cost-modeling, and equivalence-measurement techniques are needed, and that generative or learned search over the topology/model/engine space is a promising direction.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the 153x cost variance observed on one pipeline generalizes, then even a crude hill-climbing optimizer that tries a few dozen configurations would likely find order-of-magnitude improvements; the paper's own numbers make the low-hanging-fruit argument stronger than the paper states.
  • The corpus analysis suggests a concrete benchmark opportunity: a public suite of abstract workflows with measured cost/latency/accuracy for a grid of instantiations would let the community test whether the proposed Pareto-based planning actually beats manual baselines — the paper stops short of building it.
  • The 'no common operator algebra' claim implies that algebraic rewrite rules — the traditional heart of query optimization — may be largely replaced by learned search and cost prediction; a testable extension would be whether a learned planner can transfer across workflow structures without explicit rewriting rules.
  • The paper's coupling argument predicts a measurable interaction effect: the optimal engine for a deterministic task should shift when the model upstream changes, which is directly testable in existing agent frameworks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper is a vision statement arguing that multi-agent LLM pipelines require a new class of query optimizers. It motivates the argument with a ten-agent customer-support workflow and with an analysis of over 9000 (generated) multi-agent workflows, then proposes an architecture composed of a multi-objective planner, unified cost models, a multi-layer semantic/symbolic cache, a search-space manager, and a monitor. It concludes with open challenges. The paper contains no implemented system and no reported experiment in the body.

Significance. The direction is timely and potentially influential: if the thesis holds, automatic joint optimization of topology, models, and engines could substantially reduce cost and latency of agentic pipelines. The paper's strengths are a clear problem decomposition and a plausible architecture, plus a useful connection to prior cross-engine and learned-optimization work. However, the empirical motivation currently rests on claims that are either absent from the body or not reproducible, so the significance of the evidence, rather than the vision itself, is the main concern.

major comments (2)
  1. [Abstract; Sections II–III] The abstract accompanying the submission asserts a 'controlled experiment over a real-world 10-agent pipeline' with 153x cost, 5x latency, and 25% quality variance. No such experiment appears in Sections II–V. Section II is a qualitative motivating example with no measurements, and Section III is an aggregate analysis of a generated corpus, not a controlled experiment over a single real pipeline; it reports no cost/latency/quality variance numbers. Moreover, the abstract printed at the top of the paper does not contain those figures. This claim is the only quantitative support for 'extreme variance' and for the paper's central thesis that agentic optimization is fundamentally different. Please either report the experiment in sufficient detail (pipeline, search space, measurement protocol) or withdraw the claim and soften the motivation accordingly.
  2. [Section III] The workload analysis of 'over 9000 multi-agent workflows' is the other empirical pillar, but the generation procedure is not disclosed. The text says workflows were 'generated based on tens of concrete workflows' yet gives no description of the generator, the seed workflows, the sampling distribution over structures/model pools/engine pools, or any validation that the generated corpus reflects real deployments. The observed dominance of chains (34%), the concentration of model families, and narrow engine choices may be artifacts of the generator's defaults rather than evidence of systematic suboptimality in practice. The section title 'Analysis of Real Multi-Agent Workflows' and the abstract's phrase 'real deployments' are stronger than the stated provenance. Please release the corpus and generation details, or temper the claims to 'synthetic, industry-inspired workflows'.
minor comments (4)
  1. [Title/Abstract] The arXiv listing title ('Rethinking Query Optimization for Multi-Agent Systems [Vision]') and the abstract mention 'NOMA', but the paper title and body describe only an unnamed 'vision'. Harmonize the title, abstract, and body.
  2. [References] References [21] and [43] are duplicate citations of the same Quake paper; [42] and [50] are duplicate citations of the same LSM-VEC paper. Please merge.
  3. [Section V-C] In the numbered list under Challenge 4, the possible directions appear as (1), (2), (3), (5), (5); the penultimate item should be (4).
  4. [Figure 2] Figure 2's subplots would benefit from explicit axis labels and numeric annotations, particularly 2b and 2d, so that the reported percentages are readable without referring to the running text.

Circularity Check

0 steps flagged

No significant circularity: the paper is a vision/agenda with no fitted parameters or predictive derivation that reduces to its inputs. Minor self-citations exist but are not load-bearing; the abstract's unreported 'controlled experiment' is a missing-support issue, not a circularity.

full rationale

No circular derivation chain is present. Section IV-A defines abstract and executable workflows and poses the optimization problem as finding a Pareto-optimal W* for a given W and objectives O, but no fitted cost model, parameter, or empirical prediction is derived from these definitions; the paper explicitly labels the cost model and learned-estimator components as 'Possible direction' items. The self-citations to Wayang/RHEEM [13,14] and to learned cost models [16,18] are used as prior art to extend ('builds upon ideas from cross-engine query optimization [13], [14]') rather than as proof that multi-agent optimization is fundamentally different, so they are not load-bearing. The central challenges (multi-dimensional objectives, variable topology, heterogeneous data models, execution cost) are argued from the motivating example and workflow analysis, not forced by the cited works. One support issue should be flagged, but it is not circularity: the Abstract asserts 'Our controlled experiment over a real-world 10-agent pipeline reveals extreme variance (153x cost, 5x latency, 25% quality) and that optimal plans are heterogeneous configurations no user would construct manually,' yet no controlled experiment, methodology, measurements, or variance quantification appears in the body; Section III reports only an analysis of 'over 9000 multi-agent workflows (generated based on tens of concrete workflows)'. This is a reproducibility/verifiability problem for the empirical motivation, not a reduction of the paper's conclusions to its inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 2 invented entities

NOMA is a vision paper, so the ledger contains no fitted free parameters; the 153x/5x/25% figures in the abstract would be measurements, not fits, and they are unreported in the body. The claims rest on domain assumptions about the target environment: a finite, queryable agent registry; a workload corpus representative of real deployments; the premise that agent operations cannot be captured by a closed algebra; and the transferability of the authors' prior cross-engine optimization work. The invented entities are software components (NOMA, MMCache) with no implementation or falsifiable handle outside the paper.

axioms (5)
  • domain assumption A global agent registry R of available agents with functional descriptions and expected I/O exists, and the space of executable tasks is finite in practice.
    Section IV.A defines abstract workflows against registry R and asserts the practically executable subset of tasks is finite; the optimizer's search space depends on this, but no evidence is given that such a registry can be maintained in real deployments.
  • domain assumption The 9,000-workflow corpus generated from tens of concrete workflows is representative of real multi-agent deployments.
    Section III uses this corpus for all distributional findings (task distribution, engine selection, LLM usage, workflow structures). Representativeness is asserted, not validated; the abstract overstates this as 'an analysis of real deployments.'
  • domain assumption Agent operations cannot be captured by a fixed operator algebra (unlike relational algebra), justifying a fundamentally new optimizer design.
    Section I and IV.A argue the operation space is theoretically infinite; this underpins the 'fundamentally different' claim. The paper also assumes the practical subset is finite, so the practical distinction from extensible relational optimizers is asserted rather than demonstrated.
  • domain assumption Cross-engine optimization and learned cost models from the authors' prior work [13,14,16,18] transfer to settings with stochastic LLM agents.
    Section IV.B builds the unified cost model on ideas from Wayang/RHEEM and learned cost models; the transfer is stated as inspiration and is not validated in the multi-agent setting.
  • ad hoc to paper Pareto-optimal executable workflows over the joint model x engine x topology space exist and can be enumerated tractably.
    The problem statement (Section IV.A) defines the goal as finding Pareto-optimal W*; the paper itself concedes the search-space explosion is open (Challenge 3), so this assumption is necessary for the problem to be well-posed but is not established.
invented entities (2)
  • NOMA no independent evidence
    purpose: The proposed query optimization framework for multi-agent workflows, comprising a multi-objective planner, unified cost models, MMCache, search space manager, and monitor.
    Described as an envisioned architecture (Section IV.B); no implementation, benchmark, or falsifiable performance prediction is provided, so there is no handle outside the paper to test it.
  • MMCache (multi-layer multi-purpose cache) no independent evidence
    purpose: Caches intermediate results, optimized plans, and coordination strategies with exact-match symbolic and similarity-based semantic tiers.
    Section IV.B; a designed component with no implementation. Its claimed cost/benefit trade-offs (embedding cost, index maintenance) are quoted from prior work [40]–[43], not measured here.

pith-pipeline@v1.3.0-alltime-deepseek · 12677 in / 19061 out tokens · 177616 ms · 2026-08-03T17:16:54.827768+00:00 · methodology

0 comments
read the original abstract

The proliferation of large language models (LLMs) has accelerated the adoption of agent-based data pipelines. Yet current approaches remain ad hoc, relying on fixed structures, predefined LLMs, and single execution engines, without systematic optimization across heterogeneous data sources and engines. This paper presents NOMA, a query optimization framework for multi-agent data pipelines. We argue that optimizing agentic pipelines is a fundamentally different query optimization problem, with central challenges: (i)~a multi-dimensional search and objective space, where topology, model, and engine choices must be optimized jointly across latency, cost, and accuracy; (ii)~a variable pipeline topology; (iii)~the co-existence of diverse data models, leaving no common operator algebra; and (iv)~the significant cost of executing these pipelines. Our controlled experiment over a real-world 10-agent pipeline reveals extreme variance (153x cost, 5x latency, 25% quality) and that optimal plans are heterogeneous configurations no user would construct manually. Our analysis of real deployments confirms these inefficiencies are systematic. We present \system{} as an integrated optimization loop in which plan generation, cost estimation, runtime refinement, and semantic caching reinforce one another across executions, setting a community-wide research agenda on query optimization for multi-agent systems.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

171 extracted references · 12 canonical work pages · 6 internal anchors

  1. [1]

    LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead , url=

    He, Junda and Treude, Christoph and Lo, David , pages=. LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead , url=. ACM Transactions on Software Engineering and Methodology , volume=. doi:10.48550/arXiv.2404.04834 , abstractNote=

    Pith/arXiv arXiv doi:10.48550/arxiv.2404.04834

  2. [2]

    10.48550/arXiv.2307.04738 , author=

    RoCo: Dialectic Multi-Robot Collaboration with Large Language Models , url=. 10.48550/arXiv.2307.04738 , author=. doi:10.48550/arXiv.2307.04738 , abstractNote=

    Pith/arXiv arXiv doi:10.48550/arxiv.2307.04738

  3. [3]

    The Twelfth International Conference on Learning Representations (ICLR) , year=

    Building Cooperative Embodied Agents Modularly with Large Language Models , author=. The Twelfth International Conference on Learning Representations (ICLR) , year=

  4. [4]

    ICML , author=

    Improving Factuality and Reasoning in Language Models through Multiagent Debate , abstractNote=. ICML , author=. 2024 , pages=

    2024

  5. [5]

    IJCAI , author=

    Large Language Model based Multi-Agents: A Survey of Progress and Challenges , abstractNote=. IJCAI , author=

  6. [6]

    NeurIPS , author=

    DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines , abstractNote=. NeurIPS , author=

  7. [7]

    Differentiation

    Yuksekgonul, Mert and Bianchi, Federico and Boen, Joseph and Liu, Sheng and Huang, Zhi and Guestrin, Carlos and Zou, James , year=. TextGrad: Automatic “Differentiation” via Text , abstractNote=. Nature , volume=

  8. [8]

    AutoGen: Enabling Next-Gen

    Qingyun Wu and Gagan Bansal and Jieyu Zhang and Yiran Wu and Beibin Li and Erkang Zhu and Li Jiang and Xiaoyun Zhang and Shaokun Zhang and Jiale Liu and Ahmed Hassan Awadallah and Ryen W White and Doug Burger and Chi Wang , booktitle=. AutoGen: Enabling Next-Gen. 2024 , url=

    2024

  9. [9]

    The Berkeley Artificial Intelligence Research Blog , author=

    The Shift from Models to Compound AI Systems , url=. The Berkeley Artificial Intelligence Research Blog , author=

  10. [10]

    Gonzalez and Ion Stoica , booktitle=

    Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and Dacheng Li and Eric Xing and Hao Zhang and Joseph E. Gonzalez and Ion Stoica , booktitle=. Judging. 2023 , url=

    2023

  11. [11]

    Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST) , author=

    Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences , url=. Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST) , author=. 2024 , pages=. doi:10.48550/arXiv.2404.12272 , abstractNote=

    Pith/arXiv arXiv doi:10.48550/arxiv.2404.12272 2024

  12. [12]

    Yan and Haichen Shen and Meghan Cowan and Leyuan Wang and Yuwei Hu and Luis Ceze and Carlos Guestrin and Arvind Krishnamurthy , editor =

    Tianqi Chen and Thierry Moreau and Ziheng Jiang and Lianmin Zheng and Eddie Q. Yan and Haichen Shen and Meghan Cowan and Leyuan Wang and Yuwei Hu and Luis Ceze and Carlos Guestrin and Arvind Krishnamurthy , editor =. 13th. 2018 , timestamp =

    2018

  13. [13]

    CoRR , volume =

    Baoming Chang and Amin Kamali and Verena Kantere , title =. CoRR , volume =. 2025 , doi =. 2501.17414 , timestamp =

    arXiv 2025

  14. [14]

    Neurocomputing , volume =

    Online Learning: A Comprehensive Survey , author =. Neurocomputing , volume =. 2021 , publisher =. doi:10.1016/j.neucom.2021.04.112 , url =

    work page doi:10.1016/j.neucom.2021.04.112 2021

  15. [15]

    ACM Computing Surveys , volume =

    A Survey on Concept Drift Adaptation , author =. ACM Computing Surveys , volume =. 2014 , publisher =. doi:10.1145/2523813 , url =

    doi:10.1145/2523813 2014

  16. [16]

    CoRR , volume =

    Wonjun Jeong and Dongseok Kim and Taeg Keun Whangbo , title =. CoRR , volume =. 2025 , doi =. 2507.18182 , timestamp =

    Pith/arXiv arXiv 2025

  17. [17]

    arXiv preprint arXiv:2508.08053 , year=

    AdaptFlow: Adaptive Workflow Optimization via Meta-Learning , author=. arXiv preprint arXiv:2508.08053 , year=

    Pith/arXiv arXiv

  18. [18]

    Franklin and Bj

    Shaul Dar and Michael J. Franklin and Bj. Semantic Data Caching and Replacement , booktitle =. 1996 , pages =

    1996

  19. [19]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Yicheng Dang and Zhaohui Wang and Cheng Tan and Zhiwei Liu and Qiang Yang , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  20. [20]

    Proceedings of the International Conference on Learning Representations (ICLR) , year =

    Jared Ong and Yiming Lu and Sheng Shen and Yanqi Zhou and Daniel Fried and Tao Yu , title =. Proceedings of the International Conference on Learning Representations (ICLR) , year =

  21. [21]

    Proceedings of the VLDB Endowment (PVLDB) , volume =

    Zhenyu Yan and Yuxiang Zeng and Xinyi Zhang and Yifan Gao and Xuemin Lin , title =. Proceedings of the VLDB Endowment (PVLDB) , volume =. 2025 , publisher =

    2025

  22. [22]

    Proceedings of the 39th IEEE International Parallel and Distributed Processing Symposium (IPDPS) , year =

    Waris Gill and Mohamed Elidrisi and Pallavi Kalapatapu and Ammar Ahmed and Ali Anwar and Muhammad Ali Gulzar , title =. Proceedings of the 39th IEEE International Parallel and Distributed Processing Symposium (IPDPS) , year =

  23. [23]

    Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023) , year =

    Fu Bang , title =. Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023) , year =. doi:10.18653/v1/2023.nlposs-1.24 , url =

    doi:10.18653/v1/2023.nlposs-1.24 2023

  24. [24]

    2026 , eprint =

    Bespoke OLAP: Synthesizing Workload-Specific Analytical Database Systems , author =. 2026 , eprint =. doi:10.48550/arXiv.2603.02001 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.2603.02001 2026

  25. [25]

    2026 , eprint =

    GenDB: Generating Query Processing Systems with Large Language Models , author =. 2026 , eprint =. doi:10.48550/arXiv.2603.02081 , url =

    doi:10.48550/arxiv.2603.02081 2026

  26. [26]

    Proceedings of the VLDB Endowment (PVLDB) , volume=

    Learning to Optimize Join Queries With Deep Reinforcement Learning , author=. Proceedings of the VLDB Endowment (PVLDB) , volume=

  27. [27]

    Proceedings of the 38th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL) , pages=

    Automating String Processing in Spreadsheets Using Input-Output Examples , author=. Proceedings of the 38th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL) , pages=

  28. [28]

    Proceedings of the 12th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) , pages=

    Combinatorial Sketching for Finite Programs , author=. Proceedings of the 12th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) , pages=

  29. [29]

    arXiv preprint arXiv:2107.03374 , year=

    Evaluating Large Language Models Trained on Code , author=. arXiv preprint arXiv:2107.03374 , year=

    Pith/arXiv arXiv

  30. [30]

    Proceedings of the International Conference on Machine Learning (ICML) , pages=

    DreamCoder: Growing Generalizable, Interpretable Knowledge with Wake-Sleep Bayesian Program Learning , author=. Proceedings of the International Conference on Machine Learning (ICML) , pages=

  31. [31]

    Proceedings of the VLDB Endowment (PVLDB) , volume=

    Neo: A Learned Query Optimizer , author=. Proceedings of the VLDB Endowment (PVLDB) , volume=

  32. [32]

    arXiv preprint arXiv:2502.04306 , year=

    ScoreFlow: Mastering LLM Agent Workflows via Score-Based Preference Optimization , author=. arXiv preprint arXiv:2502.04306 , year=

    Pith/arXiv arXiv

  33. [33]

    arXiv preprint arXiv:2408.08435 , year=

    Automated Design of Agentic Systems (ADAS) , author=. arXiv preprint arXiv:2408.08435 , year=

    Pith/arXiv arXiv

  34. [34]

    arXiv preprint arXiv:2407.12821 , year=

    AutoFlow: Automated Workflow Generation for Large Language Model Agents , author=. arXiv preprint arXiv:2407.12821 , year=

    Pith/arXiv arXiv

  35. [35]

    arXiv preprint arXiv:2502.05957 , year=

    AutoAgent: A Fully-Automated and Zero-Code Framework for LLM Agents , author=. arXiv preprint arXiv:2502.05957 , year=

    arXiv

  36. [36]

    arXiv preprint arXiv:2308.10848 , year=

    AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviours , author=. arXiv preprint arXiv:2308.10848 , year=

    Pith/arXiv arXiv

  37. [37]

    arXiv preprint arXiv:2502.xxxxx , year=

    Palimpzest: A Declarative Framework for LLM-Powered Data Analytics , author=. arXiv preprint arXiv:2502.xxxxx , year=

  38. [38]

    CIDR Conference , year=

    CAESURA: Language Models as Multi-Modal Query Planners , author=. CIDR Conference , year=

  39. [39]

    PVLDB , year=

    LOTUS: Enabling Semantic Queries with LLMs over Tables of Unstructured and Structured Data , author=. PVLDB , year=

  40. [40]

    PVLDB , volume=

    ELEET: Efficient Learned Query Execution over Text and Tables , author=. PVLDB , volume=

  41. [41]

    2025 , url =

    OpenAI Platform: Pricing , author =. 2025 , url =

    2025

  42. [42]

    2025 , month =

    Best Open-Source Embedding Models Benchmarked and Ranked , author =. 2025 , month =

    2025

  43. [43]

    Proceedings of the 33rd International Conference on Very Large Data Bases (VLDB) , pages =

    Efficient Query Evaluation on Probabilistic Databases , author =. Proceedings of the 33rd International Conference on Very Large Data Bases (VLDB) , pages =

  44. [44]

    Foundations and Trends in Machine Learning , volume =

    Conformal Prediction: A Unified Review of Theory and New Challenges , author =. Foundations and Trends in Machine Learning , volume =. 2023 , doi =

    2023

  45. [45]

    Dalvi and Dan Suciu , title =

    Nilesh N. Dalvi and Dan Suciu , title =. 2007 , doi =

    2007

  46. [46]

    Information Systems Frontiers , volume =

    Uncertainty Management in Data and Information Systems: A Survey of Probabilistic Database Systems , author =. Information Systems Frontiers , volume =. 2007 , doi =

    2007

  47. [47]

    Frontiers Comput

    Xibin Dong and Zhiwen Yu and Wenming Cao and Yifan Shi and Qianli Ma , title =. Frontiers Comput. Sci. , volume =. 2020 , doi =

    2020

  48. [48]

    Product Quantization for Nearest Neighbor Search , journal =

    Herv. Product Quantization for Nearest Neighbor Search , journal =. 2011 , doi =

    2011

  49. [49]

    CoRR , volume =

    Victor Sanh and Lysandre Debut and Julien Chaumond and Thomas Wolf , title =. CoRR , volume =. 2019 , eprinttype =. 1910.01108 , timestamp =

    Pith/arXiv arXiv 2019

  50. [50]

    6th International Conference on Learning Representations,

    Ozan Sener and Silvio Savarese , title =. 6th International Conference on Learning Representations,. 2018 , timestamp =

    2018

  51. [51]

    CoRR , volume =

    Shurui Zhong and Dingheng Mo and Siqiang Luo , title =. CoRR , volume =. 2025 , doi =. 2505.17152 , timestamp =

    Pith/arXiv arXiv 2025

  52. [52]

    2023 , doi =

    Bingchao Wu and Yangyuxuan Kang and Daoguang Zan and Bei Guan and Yongji Wang , title =. 2023 , doi =

    2023

  53. [53]

    CoRR , volume =

    Lingxiao Kong and Cong Yang and Susanne Neufang and Oya Deniz Beyan and Zeyd Boukhers , title =. CoRR , volume =. 2025 , doi =. 2505.02579 , timestamp =

    Pith/arXiv arXiv 2025

  54. [54]

    CoRR , volume =

    Lingxiao Kong and Cong Yang and Oya Deniz Beyan and Zeyd Boukhers , title =. CoRR , volume =. 2025 , doi =. 2509.21613 , timestamp =

    arXiv 2025

  55. [55]

    9th International Conference on Learning Representations,

    Aviv Navon and Aviv Shamsian and Ethan Fetaya and Gal Chechik , title =. 9th International Conference on Learning Representations,. 2021 , timestamp =

    2021

  56. [56]

    Knowles and Weijie Zheng , editor =

    Joshua D. Knowles and Weijie Zheng , editor =. Evolutionary Multiobjective Optimization. Proceedings of the Genetic and Evolutionary Computation Conference Companion,. 2024 , doi =

    2024

  57. [57]

    Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized

    Chao Wang and Pengfei Zuo and Zhangyu Chen and Yunkai Liang and Zhou Yu and Ming. Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized. CoRR , volume =. 2025 , doi =. 2508.01989 , timestamp =

    Pith/arXiv arXiv 2025

  58. [58]

    Unlocking Efficiency in Large Language Model Inference:

    Heming Xia and Zhe Yang and Qingxiu Dong and Peiyi Wang and Yongqi Li and Tao Ge and Tianyu Liu and Wenjie Li and Zhifang Sui , editor =. Unlocking Efficiency in Large Language Model Inference:. Findings of the Association for Computational Linguistics,. 2024 , doi =

    2024

  59. [59]

    Deferred prefill for throughput maximization in

    Moonmoon Mohanty and Gautham Bolar and Preetam Patil and UmaMaheswari Devi and Felix George and Pratibha Moogi and Parimal Parag , editor =. Deferred prefill for throughput maximization in. Proceedings of the 5th Workshop on Machine Learning and Systems, EuroMLSys 2025, World Trade Center, Rotterdam, The Netherlands, 30 March 2025- 3 April 2025 , pages =....

    2025

  60. [60]

    DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving , booktitle =

    Yinmin Zhong and Shengyu Liu and Junda Chen and Jianbo Hu and Yibo Zhu and Xuanzhe Liu and Xin Jin and Hao Zhang , editor =. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving , booktitle =. 2024 , timestamp =

    2024

  61. [61]

    Ilyas and Theodoros Rekatsinas and Shivaram Venkataraman , editor =

    Jason Mohoney and Devesh Sarda and Mengze Tang and Shihabur Rahman Chowdhury and Anil Pacaci and Ihab F. Ilyas and Theodoros Rekatsinas and Shivaram Venkataraman , editor =. Quake: Adaptive Indexing for Vector Search , booktitle =. 2025 , timestamp =

    2025

  62. [62]

    Ansor: Generating High-Performance Tensor Programs for Deep Learning , booktitle =

    Lianmin Zheng and Chengfan Jia and Minmin Sun and Zhao Wu and Cody Hao Yu and Ameer Haj. Ansor: Generating High-Performance Tensor Programs for Deep Learning , booktitle =. 2020 , timestamp =

    2020

  63. [63]

    2024 , url =

    James Jie Pan and Jianguo Wang and Guoliang Li , title =. 2024 , url =. doi:10.1007/S00778-024-00864-X , timestamp =

    doi:10.1007/s00778-024-00864-x 2024

  64. [64]

    Ilyas and Theodoros Rekatsinas and Shivaram Venkataraman , editor =

    Jason Mohoney and Devesh Sarda and Mengze Tang and Shihabur Rahman Chowdhury and Anil Pacaci and Ihab F. Ilyas and Theodoros Rekatsinas and Shivaram Venkataraman , editor =. Quake: Adaptive Indexing for Vector Search , booktitle =. 2025 , url =

    2025

  65. [65]

    CoRR , volume =

    Shurui Zhong and Dingheng Mo and Siqiang Luo , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2505.17152 , eprinttype =. 2505.17152 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.17152 2025

  66. [66]

    CoRR , volume =

    Sajal Regmi and Chetan Phakami Pun , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2411.05276 , eprinttype =. 2411.05276 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2411.05276 2024

  67. [67]

    Jianguo Wang and Xiaomeng Yi and Rentong Guo and Hai Jin and Peng Xu and Shengjun Li and Xiangyu Wang and Xiangzhou Guo and Chengming Li and Xiaohai Xu and Kun Yu and Yuxing Yuan and Yinghao Zou and Jiquan Long and Yudong Cai and Zhenxiang Li and Zhifeng Zhang and Yihua Mo and Jun Gu and Ruiyi Jiang and Yi Wei and Charles Xie , editor =. Milvus:. 2021 , u...

    arXiv 2021

  68. [68]

    The Twelfth International Conference on Learning Representations (ICLR) , year=

    Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models , author=. The Twelfth International Conference on Learning Representations (ICLR) , year=

  69. [69]

    The Twelfth International Conference on Learning Representations (ICLR) , year=

    Evaluating Large Language Models at Evaluating Instruction Following , author=. The Twelfth International Conference on Learning Representations (ICLR) , year=

  70. [70]

    10.48550/arXiv.2411.15594 , author=

    A Survey on LLM-as-a-Judge , url=. 10.48550/arXiv.2411.15594 , author=. doi:10.48550/arXiv.2411.15594 , abstractNote=

    Pith/arXiv arXiv doi:10.48550/arxiv.2411.15594

  71. [71]

    Towards Modeling Human-Agentic Collaborative Workflows: A BPMN Extension , url=

    Ait, Adem and Izquierdo, Javier Luis Cánovas and Cabot, Jordi , year=. Towards Modeling Human-Agentic Collaborative Workflows: A BPMN Extension , url=. doi:10.48550/arXiv.2412.05958 , abstractNote=

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.05958

  72. [72]

    WorkTeam: Constructing Workflows from Natural Language with Multi-Agents

    Liu, Hanchao and Li, Rongjun and Xiong, Weimin and Zhou, Ziyu and Peng, Wei , year=. WorkTeam: Constructing Workflows from Natural Language with Multi-Agents , url=. doi:10.48550/arXiv.2503.22473 , abstractNote=

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2503.22473

  73. [73]

    AIAP: A No-Code Workflow Builder for Non-Experts with Natural Language and Multi-Agent Collaboration

    An, Hyunjn and Kim, Yongwon and Seo, Wonduk and Park, Joonil and Kang, Daye and Oh, Changhoon and Kim, Dokyun and Lee, Seunghyun , year=. AIAP: A No-Code Workflow Builder for Non-Experts with Natural Language and Multi-Agent Collaboration , url=. doi:10.48550/arXiv.2508.02470 , abstractNote=

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2508.02470

  74. [74]

    ICLR , author=

    MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework , abstractNote=. ICLR , author=

  75. [75]

    and Parameswaran, Aditya G

    Liu, Shu and Ponnapalli, Soujanya and Shankar, Shreya and Zeighami, Sepanta and Zhu, Alan and Agarwal, Shubham and Chen, Ruiqi and Suwito, Samion and Yuan, Shuo and Stoica, Ion and Zaharia, Matei and Cheung, Alvin and Crooks, Natacha and Gonzalez, Joseph E. and Parameswaran, Aditya G. , year=. Supporting Our AI Overlords: Redesigning Data Systems to be Ag...

    doi:10.48550/arxiv.2509.00997

  76. [76]

    Stage: Query Execution Time Prediction in Amazon Redshift

    Stage: Query Execution Time Prediction in Amazon Redshift , url=. Companion of the 2024 International Conference on Management of Data , author=. doi:10.48550/arXiv.2403.02286 , abstractNote=

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2403.02286 2024

  77. [77]

    10.48550/arXiv.2506.02153 , author=

    Small Language Models are the Future of Agentic AI , url=. 10.48550/arXiv.2506.02153 , author=. doi:10.48550/arXiv.2506.02153 , abstractNote=

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.02153

  78. [78]

    Bootstrapping Learned Cost Models with Synthetic SQL Queries

    Bootstrapping Learned Cost Models with Synthetic SQL Queries , url=. AI for Databases Workshop (co-located with VLDB) , author=. doi:10.48550/arXiv.2508.19807 , abstractNote=

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2508.19807

  79. [79]

    Adapting LLMs for Structured Natural Language API Integration , url=

    Chan, Robin and Mirylenka, Katsiaryna and Gschwind, Thomas and Miksovic, Christoph and Scotton, Paolo and Toniato, Enrico and Labbi, Abdel , editor=. Adapting LLMs for Structured Natural Language API Integration , url=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track , publisher=. 2024 , month=nov, pa...

    work page doi:10.18653/v1/2024.emnlp-industry.74 2024

  80. [80]

    Are Bias Evaluation Methods Biased ?

    Berrayana, Lina and Rooney, Sean and Garcés-Erice, Luis and Giurgiu, Ioana , year=. Are Bias Evaluation Methods Biased ? , url=. doi:10.48550/arXiv.2506.17111 , abstractNote=

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2506.17111

Showing first 80 references.