REVIEW 26 cited by
Semantic Operators: A Declarative Model for Rich, AI-based Data Processing
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
The semantic capabilities of large language models (LLMs) have the potential to enable rich analytics and reasoning over vast knowledge corpora. Unfortunately, existing systems either empirically optimize expensive LLM-powered operations with no performance guarantees, or serve a limited set of row-wise LLM operations, providing limited robustness, expressiveness and usability. We introduce semantic operators, the first formalism for declarative and general-purpose AI-based transformations based on natural language specifications (e.g., filtering, sorting, joining or aggregating records using natural language criteria). Each operator opens a rich space for execution plans, similar to relational operators. Our model specifies the expected behavior of each operator with a high-quality gold algorithm, and we develop an optimization framework that reduces cost, while providing accuracy guarantees with respect to a gold algorithm. Using this approach, we propose several novel optimizations to accelerate semantic filtering, joining, group-by and top-k operations by up to $1,000\times$. We implement semantic operators in the LOTUS system and demonstrate LOTUS' effectiveness on real, bulk-semantic processing applications, including fact-checking, biomedical multi-label classification, search, and topic analysis. We show that the semantic operator model is expressive, capturing state-of-the-art AI pipelines in a few operator calls, and making it easy to express new pipelines that match or exceed quality of recent LLM-based analytic systems by up to $170\%$, while offering accuracy guarantees. Overall, LOTUS programs match or exceed the accuracy of state-of-the-art AI pipelines for each task while running up to $3.6\times$ faster than the highest-quality baselines. LOTUS is publicly available at https://github.com/lotus-data/lotus.
Forward citations
Cited by 26 Pith papers
-
Confining Nondeterminism: AI-Driven Research Systems as DBMSs for Reliable, Non-Wasteful, Transparent, and Collaborative Research [Vision]
Confine the LLM to compiling plan edits over a deterministic, versioned dataflow executor that never calls the model, so research guarantees hold by construction.
-
AnnoRetrieve: Efficient Structured Retrieval for Unstructured Document Analysis
AnnoRetrieve uses auto-generated structured schemas and queries to retrieve information from unstructured documents more efficiently and accurately than embedding-based methods.
-
Cut Costs, Not Accuracy: LLM-Powered Data Processing with Guarantees
BARGAIN uses betting-based anytime-valid tests and adaptive, target-aware sampling to set model-cascade thresholds, delivering non-asymptotic quality guarantees and up to 86% greater cost savings than SUPG.
-
Structure then Query: Enabling Precise Analytical Queries over Unstructured Documents
AnnoIndex reaches 0.87 average F1 on three document corpora by building an offline annotation index with induced schemas and a progressive SQL-based query engine that reuses extracted values.
-
Spider 2.0-AIFunc: Extending Real-World Text-to-SQL to AI-Native SQL Workflows
Spider 2.0-AIFunc is a 465-instance benchmark for evaluating text-to-SQL systems on queries that incorporate Snowflake Cortex AI functions, with evaluations of ten models showing proprietary models reach 67-70% accuracy.
-
Natural Language to What? A Vision for Intermediate Representations in NL-to-X Querying
NLIQ frames natural language querying around target adequacy to classify when the output representation must be constructed, treating intermediate representations as first-class semantic objects rather than mere imple...
-
The Case for Instance-Optimized LLMs in OLAP Databases
IOLM-DB compresses a general-purpose LLM on a per-query basis to speed up LLM-invoking OLAP queries, reporting throughput gains up to 3.31x and smaller models in a three-workload prototype.
-
Parachute: Single-Pass Bi-Directional Information Passing
Parachute stores precomputed join-induced fingerprint columns on FK-tables to enable single-pass bi-directional sideways information passing, speeding up JOB by up to 1.54x over DuckDB v1.2 at 15% extra space.
-
SQLens: An End-to-End Framework for Error Detection and Correction in Text-to-SQL
SQLens uses weak supervision over database and LLM error signals to detect clause-level semantic errors in text-to-SQL and fixes them iteratively, improving execution accuracy on BIRD and Spider.
-
RelationalFactQA: A Benchmark for Evaluating Tabular Fact Retrieval from Large Language Models
RelationalFactQA shows that LLMs are much worse at retrieving facts as multi-record tables than as single answers, with the best model reaching only 24.7% tuple similarity.
-
Steering Semantic Data Processing With DocWrangler
An IDE for LLM-powered text data processing, with user studies showing that people convert open-ended operations into structured classifiers and use vague prompts to explore their data.
-
CoddLLM: Empowering Large Language Models for Data Analytics
CoddLLM, a Mistral-NeMo-12B model post-trained on synthetic analytics instruction data, outperforms GPT-4o on the paper's table-selection average and improves text-to-SQL over its base model, with the largest gains on...
-
Fundamental Challenges in Evaluating Text2SQL Solutions and Detecting Their Limitations
Aggregate Text2SQL benchmark numbers are distorted by ambiguous single labels and by the SQL-equivalence match functions, a problem the paper organizes into a taxonomy with concrete Spider examples.
-
TQA-Bench: Evaluating LLMs for Multi-Table Question Answering
TQA-Bench is a multi-table QA benchmark with 8K-64K contexts and symbolic question extensions; tests on 22 LLMs show accuracy declines with context length.
-
Variable Extraction for Model Recovery in Scientific Literature
LLM-based variable extraction outperforms rule-based methods on a new 22-paper benchmark, with best F1 around 0.53 to 0.64, but all systems remain far from solved.
-
Monte Carlo Tree Search for Table-to-Multimodal Report Generation
MCTS-Report applies Monte Carlo Tree Search to multimodal table-to-report generation, reaching a 77.9 overall score on the new MMRBench benchmark, but its evaluation relies on a single, unvalidated LLM judge.
-
Deep Research is the New Analytics System: Towards Building the Runtime for AI-Driven Analytics
A prototype combining Deep Research planning with Palimpzest-style semantic operator optimization beats open-code agents on two unstructured analytics queries.
-
SABER: A SQL-Compatible Semantic Document Processing System Based on Extended Relational Algebra
SABER defines semantic counterparts of relational algebra operators, exposes them as SQL UDFs, and maps them onto LOTUS, DocETL, and Palimpzest, yielding portable semantic queries.
-
Research Challenges in Relational Database Management Systems for LLM Queries
Open-source SQL-plus-LLM database integrations fail on most representative LLM queries and run orders of magnitude slower than enterprise APIs, due to missing structured outputs, poor batching, and plan-unaware LLM calls.
-
Metadata Management for AI-Augmented Data Workflows
TableVault is a metadata governance framework that records ingestion events, operation status, execution parameters, and lineage for human-AI data workflows, demonstrated on a document classification case study.
-
QUEST: Query Optimization in Unstructured Document Analysis
QUEST reduces LLM extraction cost in unstructured document analytics by retrieving only relevant segments via a two-level index and by generating per-document filter and join execution plans during query execution.
-
CHASE: A Native Relational Database for Hybrid Queries on Structured and Unstructured Data
CHASE natively integrates ANN vector search into a compiled relational engine, cuts redundant similarity computation in hybrid queries, and reports up to 7,500x speedups.
-
A Survey on Open Dataset Search in the LLM Era: Retrospectives and Perspectives
A structured review of open dataset search across tabular, spatial, JSON, graph, and vector data, plus the two-way relationship with LLMs.
-
TableVault: Managing Dynamic Data Collections for LLM-Augmented Workflows
TableVault describes a system design for managing versioned, reproducible dataframe collections in LLM-augmented workflows, but it ships no implementation or evaluation.
-
PalimpChat: Declarative and Interactive AI analytics
A natural-language chat interface built on the Palimpzest declarative framework lets non-experts create and run AI data pipelines, demonstrated on a colorectal cancer literature search.
-
Querying Databases with Function Calling
A new tool definition and synthetic benchmark show top LLMs can format database query calls via function calling, with the best models scoring around 74% exact-match accuracy.
Discussion (0). Continue with ORCID to comment.