REVIEW 13 cited by
DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Analyzing unstructured data has been a persistent challenge in data processing. Large Language Models (LLMs) have shown promise in this regard, leading to recent proposals for declarative frameworks for LLM-powered processing of unstructured data. However, these frameworks focus on reducing cost when executing user-specified operations using LLMs, rather than improving accuracy, executing most operations as-is (in a single LLM call). This is problematic for complex tasks and data, where LLM outputs for user-defined operations are often inaccurate, even with optimized prompts. For example, an LLM may struggle to identify {\em all} instances of specific clauses, like force majeure or indemnification, in lengthy legal documents, requiring decomposition of the data, the task, or both. We present DocETL, a system that optimizes complex document processing pipelines, while accounting for LLM shortcomings. DocETL offers a declarative interface for users to define such pipelines and uses an agent-based approach to automatically optimize them, leveraging novel agent-based rewrites (that we call rewrite directives), as well as an optimization and evaluation framework. We introduce (i) logical rewriting of pipelines, tailored for LLM-based tasks, (ii) an agent-guided plan evaluation mechanism that synthesizes and orchestrates task-specific validation prompts, and (iii) an optimization algorithm that efficiently finds promising plans, considering the latencies of agent-based plan generation and evaluation. Our evaluation on four different unstructured document analysis tasks demonstrates that DocETL finds plans with outputs that are 25 to 80% more accurate than well-engineered baselines, addressing a critical gap in unstructured data analysis. DocETL is open-source at docetl.org, and as of March 2025, has amassed over 1.7k GitHub Stars, with users spanning a variety of domains.
Forward citations
Cited by 13 Pith papers
-
AnnoRetrieve: Efficient Structured Retrieval for Unstructured Document Analysis
AnnoRetrieve uses auto-generated structured schemas and queries to retrieve information from unstructured documents more efficiently and accurately than embedding-based methods.
-
Cut Costs, Not Accuracy: LLM-Powered Data Processing with Guarantees
BARGAIN uses betting-based anytime-valid tests and adaptive, target-aware sampling to set model-cascade thresholds, delivering non-asymptotic quality guarantees and up to 86% greater cost savings than SUPG.
-
Beyond Tables: Doc2DB-Bench for Relationally Faithful Document-to-Database Construction
Doc2DB-Bench provides 203 long-document, database pairs to test whether LLMs can reconstruct multi-table relational databases with correct keys, relationships, and constraints.
-
ASMR: Agentic Schema Generation for Ship Maintenance Report Writing
ASMR extracts concepts with an LLM, clusters them into candidate fields, then uses RL to select compact non-redundant schemas for each ship-report form type.
-
PolicyPulse: LLM-Synthesis Tool for Policy Researchers
PolicyPulse uses GPT-4 to turn Reddit discussions into policy-relevant themes and quotes, covering 73-84% of themes in two authoritative reports and helping 11 policy researchers spark research.
-
RelationalFactQA: A Benchmark for Evaluating Tabular Fact Retrieval from Large Language Models
RelationalFactQA shows that LLMs are much worse at retrieving facts as multi-record tables than as single answers, with the best model reaching only 24.7% tuple similarity.
-
HVM-GraphRAG: Holistic-View Multimodal Graph Retrieval-Augmented Generation on Complex Document
A holistic-view multimodal GraphRAG that conflict-resolves concept indices and retrieves via compact concept anchors plus modality-grouped evidence, beating strong baselines on three complex-document QA sets.
-
Deep Research is the New Analytics System: Towards Building the Runtime for AI-Driven Analytics
A prototype combining Deep Research planning with Palimpzest-style semantic operator optimization beats open-code agents on two unstructured analytics queries.
-
SABER: A SQL-Compatible Semantic Document Processing System Based on Extended Relational Algebra
SABER defines semantic counterparts of relational algebra operators, exposes them as SQL UDFs, and maps them onto LOTUS, DocETL, and Palimpzest, yielding portable semantic queries.
-
Research Challenges in Relational Database Management Systems for LLM Queries
Open-source SQL-plus-LLM database integrations fail on most representative LLM queries and run orders of magnitude slower than enterprise APIs, due to missing structured outputs, poor batching, and plan-unaware LLM calls.
-
Metadata Management for AI-Augmented Data Workflows
TableVault is a metadata governance framework that records ingestion events, operation status, execution parameters, and lineage for human-AI data workflows, demonstrated on a document classification case study.
-
TableVault: Managing Dynamic Data Collections for LLM-Augmented Workflows
TableVault describes a system design for managing versioned, reproducible dataframe collections in LLM-augmented workflows, but it ships no implementation or evaluation.
-
PalimpChat: Declarative and Interactive AI analytics
A natural-language chat interface built on the Palimpzest declarative framework lets non-experts create and run AI data pipelines, demonstrated on a colorectal cancer literature search.
Discussion (0). Continue with ORCID to comment.