{"work":{"id":"d3d9805e-cd99-435d-9eee-3dd96fe44e78","openalex_id":null,"doi":null,"arxiv_id":"2410.12189","raw_key":null,"title":"DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing","authors":null,"authors_text":"S","year":2024,"venue":"cs.DB","abstract":"Analyzing unstructured data has been a persistent challenge in data processing. Large Language Models (LLMs) have shown promise in this regard, leading to recent proposals for declarative frameworks for LLM-powered processing of unstructured data. However, these frameworks focus on reducing cost when executing user-specified operations using LLMs, rather than improving accuracy, executing most operations as-is (in a single LLM call). This is problematic for complex tasks and data, where LLM outputs for user-defined operations are often inaccurate, even with optimized prompts. For example, an LLM may struggle to identify {\\em all} instances of specific clauses, like force majeure or indemnification, in lengthy legal documents, requiring decomposition of the data, the task, or both.\n  We present DocETL, a system that optimizes complex document processing pipelines, while accounting for LLM shortcomings. DocETL offers a declarative interface for users to define such pipelines and uses an agent-based approach to automatically optimize them, leveraging novel agent-based rewrites (that we call rewrite directives), as well as an optimization and evaluation framework. We introduce (i) logical rewriting of pipelines, tailored for LLM-based tasks, (ii) an agent-guided plan evaluation mechanism that synthesizes and orchestrates task-specific validation prompts, and (iii) an optimization algorithm that efficiently finds promising plans, considering the latencies of agent-based plan generation and evaluation. Our evaluation on four different unstructured document analysis tasks demonstrates that DocETL finds plans with outputs that are 25 to 80% more accurate than well-engineered baselines, addressing a critical gap in unstructured data analysis. DocETL is open-source at docetl.org, and as of March 2025, has amassed over 1.7k GitHub Stars, with users spanning a variety of domains.","external_url":"https://arxiv.org/abs/2410.12189","cited_by_count":null,"metadata_source":"pith","metadata_fetched_at":"2026-07-10T11:57:03.407150+00:00","pith_arxiv_id":"2410.12189","created_at":"2026-05-09T22:49:16.108571+00:00","updated_at":"2026-07-10T11:57:03.407150+00:00","title_quality_ok":true,"display_title":"Parameswaran, and Eugene Wu","render_title":"Parameswaran, and Eugene Wu"},"hub":{"state":{"work_id":"d3d9805e-cd99-435d-9eee-3dd96fe44e78","tier":"hub","tier_reason":"10+ Pith inbound or 1,000+ external citations","pith_inbound_count":20,"external_cited_by_count":null,"distinct_field_count":5,"first_pith_cited_at":"2025-08-07T03:49:56+00:00","last_pith_cited_at":"2026-07-09T07:25:28+00:00","author_build_status":"not_needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"not_needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-08-04T19:59:17.642239+00:00","tier_text":"hub"},"tier":"hub","role_counts":[{"context_role":"background","n":3}],"polarity_counts":[{"context_polarity":"background","n":3}],"runs":{},"summary":{},"graph":{},"authors":[]}}